[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:33:35 dist-test-slave-77v9 krb5kdc[17324](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:33:35 dist-test-slave-77v9 krb5kdc[17324](info): set up 2 sockets
Aug 10 22:33:35 dist-test-slave-77v9 krb5kdc[17324](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260810 22:33:36.609627 17317 mini_kdc.cc:121] Time spent starting KDC: real 1.128s user 0.011s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/krb5kdc/test-user.keytab.
W20260810 22:33:36.885181 17317 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.184s user 0.000s sys 0.002s
Aug 10 22:33:36 dist-test-slave-77v9 krb5kdc[17324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:33:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:33:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.233.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab.
Entry for principal kudu/127.16.233.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab.
W20260810 22:33:38.031558 17317 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.16.233.126: real 1.117s user 0.002s sys 0.003s
I20260810 22:33:38.033130 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:39883
--webserver_interface=127.16.233.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab
--principal=kudu/127.16.233.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:41291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.126:39883
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:33:38.181444 17339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:33:38.181756 17339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:33:38.181800 17339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:33:38.186100 17339 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:33:38.186172 17339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:33:38.186192 17339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:33:38.186209 17339 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:33:38.186224 17339 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:33:38.193647 17339 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:41291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.126:39883
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab
--principal=kudu/127.16.233.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:39883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17339
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:33:38.195806 17339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:33:38.197576 17339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:33:38.212862 17344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:33:38.212839 17347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:33:38.213120 17345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:33:38.214071 17339 server_base.cc:1034] running on GCE node
I20260810 22:33:38.215145 17339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:33:38.217168 17339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:33:38.218392 17339 hybrid_clock.cc:648] HybridClock initialized: now 1786401218218363 us; error 81 us; skew 500 ppm
Aug 10 22:33:38 dist-test-slave-77v9 krb5kdc[17324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401218, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:33:38.222507 17339 init.cc:377] Logged in from keytab as kudu/127.16.233.126@KRBTEST.COM (short username kudu)
I20260810 22:33:38.223925 17339 webserver.cc:460] Webserver started at http://127.16.233.126:36599/ using document root <none> and password file <none>
I20260810 22:33:38.224941 17339 fs_manager.cc:359] Metadata directory not provided
I20260810 22:33:38.225073 17339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:33:38.225510 17339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:33:38.228648 17339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/minicluster-data/master-0/data/instance:
uuid: "7222cd1180014f07baa7ceb0d9ceb91e"
format_stamp: "Formatted at 2026-08-10 22:33:38 on dist-test-slave-77v9"
server_key: "66e77be4194dc521bb17d723d334d0d5"
server_key_iv: "e175a86172b33f5029d9bbf9044c96bc"
server_key_version: "encryptionkey@0"
I20260810 22:33:38.229885 17339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/minicluster-data/master-0/wal/instance:
uuid: "7222cd1180014f07baa7ceb0d9ceb91e"
format_stamp: "Formatted at 2026-08-10 22:33:38 on dist-test-slave-77v9"
server_key: "66e77be4194dc521bb17d723d334d0d5"
server_key_iv: "e175a86172b33f5029d9bbf9044c96bc"
server_key_version: "encryptionkey@0"
I20260810 22:33:38.235587 17339 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 22:33:38.239705 17356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:38.241856 17339 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 22:33:38.242105 17339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/minicluster-data/master-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/minicluster-data/master-0/wal
uuid: "7222cd1180014f07baa7ceb0d9ceb91e"
format_stamp: "Formatted at 2026-08-10 22:33:38 on dist-test-slave-77v9"
server_key: "66e77be4194dc521bb17d723d334d0d5"
server_key_iv: "e175a86172b33f5029d9bbf9044c96bc"
server_key_version: "encryptionkey@0"
I20260810 22:33:38.242326 17339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:33:38.258543 17339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:33:38.263074 17339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:33:38.263450 17339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:33:38.278728 17339 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.126:39883
I20260810 22:33:38.278767 17414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.126:39883 every 8 connection(s)
I20260810 22:33:38.280232 17339 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
I20260810 22:33:38.284404 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 17339
I20260810 22:33:38.284659 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/minicluster-data/master-0/wal/instance
I20260810 22:33:38.286435 17415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:33:38.287045 17317 external_mini_cluster.cc:1428] Setting key 4ccd51ce3367ef0b913dfd09f91efaff
I20260810 22:33:38.298242 17415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7222cd1180014f07baa7ceb0d9ceb91e: Bootstrap starting.
I20260810 22:33:38.303212 17415 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7222cd1180014f07baa7ceb0d9ceb91e: Neither blocks nor log segments found. Creating new log.
I20260810 22:33:38.305039 17415 log.cc:824] T 00000000000000000000000000000000 P 7222cd1180014f07baa7ceb0d9ceb91e: Log is configured to *not* fsync() on all Append() calls
I20260810 22:33:38.308580 17415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7222cd1180014f07baa7ceb0d9ceb91e: No bootstrap required, opened a new log
Aug 10 22:33:38 dist-test-slave-77v9 krb5kdc[17324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:33:38.315227 17415 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7222cd1180014f07baa7ceb0d9ceb91e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7222cd1180014f07baa7ceb0d9ceb91e" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39883 } }
I20260810 22:33:38.315696 17415 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7222cd1180014f07baa7ceb0d9ceb91e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:33:38.315819 17415 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7222cd1180014f07baa7ceb0d9ceb91e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7222cd1180014f07baa7ceb0d9ceb91e, State: Initialized, Role: FOLLOWER
I20260810 22:33:38.316676 17415 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7222cd1180014f07baa7ceb0d9ceb91e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7222cd1180014f07baa7ceb0d9ceb91e" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39883 } }
I20260810 22:33:38.316936 17415 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7222cd1180014f07baa7ceb0d9ceb91e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:33:38.317077 17415 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7222cd1180014f07baa7ceb0d9ceb91e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:33:38.317245 17415 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7222cd1180014f07baa7ceb0d9ceb91e [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:33:38.318290 17415 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7222cd1180014f07baa7ceb0d9ceb91e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7222cd1180014f07baa7ceb0d9ceb91e" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39883 } }
I20260810 22:33:38.318982 17415 leader_election.cc:302] T 00000000000000000000000000000000 P 7222cd1180014f07baa7ceb0d9ceb91e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7222cd1180014f07baa7ceb0d9ceb91e; no voters:
I20260810 22:33:38.319490 17415 leader_election.cc:288] T 00000000000000000000000000000000 P 7222cd1180014f07baa7ceb0d9ceb91e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:33:38.319692 17420 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7222cd1180014f07baa7ceb0d9ceb91e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:33:38.319976 17420 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7222cd1180014f07baa7ceb0d9ceb91e [term 1 LEADER]: Becoming Leader. State: Replica: 7222cd1180014f07baa7ceb0d9ceb91e, State: Running, Role: LEADER
I20260810 22:33:38.320533 17420 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7222cd1180014f07baa7ceb0d9ceb91e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7222cd1180014f07baa7ceb0d9ceb91e" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39883 } }
I20260810 22:33:38.321300 17415 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7222cd1180014f07baa7ceb0d9ceb91e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:33:38.323901 17418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:38.295646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59080 (local address 127.16.233.126:39883)
0810 22:33:38.296138 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:38.296153 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:38.296227 (+ 74us) server_negotiation.cc:407] Connection header received
0810 22:33:38.301129 (+ 4902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:38.301148 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:38.301612 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:38.302121 (+ 509us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:38.306253 (+ 4132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:38.307500 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:38.309759 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:38.310295 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:38.310455 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:38.316213 (+ 5758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:38.316262 (+ 49us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:38.316299 (+ 37us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:38.316339 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:38.318740 (+ 2401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:38.319623 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:38.319630 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:38.319636 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:38.319830 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:38.320422 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:38.320425 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:38.320427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:38.321161 (+ 734us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:38.321443 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:38.322292 (+ 849us) server_negotiation.cc:299] Negotiation successful
0810 22:33:38.322646 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":140,"threads_started":1}
I20260810 22:33:38.324308 17421 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7222cd1180014f07baa7ceb0d9ceb91e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7222cd1180014f07baa7ceb0d9ceb91e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7222cd1180014f07baa7ceb0d9ceb91e" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39883 } } }
I20260810 22:33:38.324678 17421 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7222cd1180014f07baa7ceb0d9ceb91e [sys.catalog]: This master's current role is: LEADER
I20260810 22:33:38.325543 17423 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7222cd1180014f07baa7ceb0d9ceb91e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7222cd1180014f07baa7ceb0d9ceb91e. Latest consensus state: current_term: 1 leader_uuid: "7222cd1180014f07baa7ceb0d9ceb91e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7222cd1180014f07baa7ceb0d9ceb91e" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39883 } } }
I20260810 22:33:38.325744 17423 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7222cd1180014f07baa7ceb0d9ceb91e [sys.catalog]: This master's current role is: LEADER
I20260810 22:33:38.325732 17438 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:33:38.330897 17438 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:33:38.339690 17438 catalog_manager.cc:1347] Generated new cluster ID: eeefd409ed074fc59b9ae5b8d56fae40
I20260810 22:33:38.339833 17438 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:33:38.362316 17438 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:33:38.363880 17438 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:33:38.376555 17438 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7222cd1180014f07baa7ceb0d9ceb91e: Generated new TSK 0
I20260810 22:33:38.377792 17438 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.233.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab.
Entry for principal kudu/127.16.233.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab.
I20260810 22:33:38.417399 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.65:0
--local_ip_for_outbound_sockets=127.16.233.65
--webserver_interface=127.16.233.65
--webserver_port=0
--tserver_master_addrs=127.16.233.126:39883
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab
--principal=kudu/127.16.233.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:41291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:33:38.575747 17445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:33:38.576208 17445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:33:38.576309 17445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:33:38.580551 17445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:33:38.580667 17445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:33:38.580771 17445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.65
I20260810 22:33:38.586697 17445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:41291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab
--principal=kudu/127.16.233.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.233.65
--webserver_port=0
--tserver_master_addrs=127.16.233.126:39883
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.65
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:33:38.588003 17445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:33:38.589174 17445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:33:38.600754 17453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:33:38.600760 17451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:33:38.600796 17450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:33:38.601039 17445 server_base.cc:1034] running on GCE node
I20260810 22:33:38.601867 17445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:33:38.602617 17445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:33:38.603842 17445 hybrid_clock.cc:648] HybridClock initialized: now 1786401218603808 us; error 50 us; skew 500 ppm
Aug 10 22:33:38 dist-test-slave-77v9 krb5kdc[17324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401218, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:33:38.607667 17445 init.cc:377] Logged in from keytab as kudu/127.16.233.65@KRBTEST.COM (short username kudu)
I20260810 22:33:38.608755 17445 webserver.cc:460] Webserver started at http://127.16.233.65:38687/ using document root <none> and password file <none>
I20260810 22:33:38.609380 17445 fs_manager.cc:359] Metadata directory not provided
I20260810 22:33:38.609483 17445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:33:38.609730 17445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:33:38.611959 17445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/minicluster-data/ts-0/data/instance:
uuid: "cae1a1a1324e4f23878843e810869118"
format_stamp: "Formatted at 2026-08-10 22:33:38 on dist-test-slave-77v9"
server_key: "5cbb8d6bc8db9f0830779b1804d619fb"
server_key_iv: "00e2ee95317d030fbba52cce842b7d00"
server_key_version: "encryptionkey@0"
I20260810 22:33:38.612671 17445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/minicluster-data/ts-0/wal/instance:
uuid: "cae1a1a1324e4f23878843e810869118"
format_stamp: "Formatted at 2026-08-10 22:33:38 on dist-test-slave-77v9"
server_key: "5cbb8d6bc8db9f0830779b1804d619fb"
server_key_iv: "00e2ee95317d030fbba52cce842b7d00"
server_key_version: "encryptionkey@0"
I20260810 22:33:38.617352 17445 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 22:33:38.620124 17460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:38.621507 17445 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:33:38.621649 17445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/minicluster-data/ts-0/wal
uuid: "cae1a1a1324e4f23878843e810869118"
format_stamp: "Formatted at 2026-08-10 22:33:38 on dist-test-slave-77v9"
server_key: "5cbb8d6bc8db9f0830779b1804d619fb"
server_key_iv: "00e2ee95317d030fbba52cce842b7d00"
server_key_version: "encryptionkey@0"
I20260810 22:33:38.621767 17445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:33:38.640281 17445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:33:38.644062 17445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:33:38.644297 17445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:33:38.645458 17445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:33:38.647209 17445 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:33:38.647279 17445 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:38.647385 17445 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:33:38.647418 17445 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:38.665593 17445 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.65:33211
I20260810 22:33:38.665817 17585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.65:33211 every 8 connection(s)
I20260810 22:33:38.666672 17445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
I20260810 22:33:38.667835 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 17445
I20260810 22:33:38.667959 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/minicluster-data/ts-0/wal/instance
I20260810 22:33:38.668212 17317 external_mini_cluster.cc:1428] Setting key 7691a741e2f1b5221a5db1322efc33d1
Aug 10 22:33:38 dist-test-slave-77v9 krb5kdc[17324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401218, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.65@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.233.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.66@KRBTEST.COM" created.
I20260810 22:33:38.686105 17418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:38.671289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:44551 (local address 127.16.233.126:39883)
0810 22:33:38.671455 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:38.671461 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:38.671479 (+ 18us) server_negotiation.cc:407] Connection header received
0810 22:33:38.672563 (+ 1084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:38.672567 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:38.672626 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:38.672721 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:38.674707 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:38.675308 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:38.677379 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:38.677646 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:38.677760 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:38.681381 (+ 3621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:38.681384 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:38.681386 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:38.681416 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:38.683392 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:38.684010 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:38.684013 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:38.684014 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:38.684113 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:38.684566 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:38.684570 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:38.684572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:38.684888 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:38.684990 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:38.685638 (+ 648us) server_negotiation.cc:299] Negotiation successful
0810 22:33:38.685883 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260810 22:33:38.686908 17589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:38.670039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39883 (local address 127.16.233.65:44551)
0810 22:33:38.671218 (+ 1179us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:38.671273 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:38.672330 (+ 1057us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:38.672932 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:38.672939 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:38.673652 (+ 713us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:33:38.674512 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:38.674538 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:38.675472 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:38.675479 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:38.677211 (+ 1732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:38.677228 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:38.677874 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:38.677883 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:38.678160 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:38.678964 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:33:38.678994 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:33:38.681115 (+ 2121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:33:38.683558 (+ 2443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:38.683563 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:38.683578 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:38.683883 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:38.684201 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:38.684203 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:38.684204 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:38.684400 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:38.685001 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:33:38.685006 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:33:38.685337 (+ 331us) client_negotiation.cc:769] Sending connection context
0810 22:33:38.685566 (+ 229us) client_negotiation.cc:240] Negotiation successful
0810 22:33:38.685928 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":927,"thread_start_us":274,"threads_started":1}
I20260810 22:33:38.688654 17586 heartbeater.cc:347] Connected to a master server at 127.16.233.126:39883
I20260810 22:33:38.688982 17586 heartbeater.cc:464] Registering TS with master...
I20260810 22:33:38.689558 17586 heartbeater.cc:511] Master 127.16.233.126:39883 requested a full tablet report, sending...
I20260810 22:33:38.691957 17374 ts_manager.cc:194] Registered new tserver with Master: cae1a1a1324e4f23878843e810869118 (127.16.233.65:33211)
I20260810 22:33:38.693562 17374 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.65@KRBTEST.COM'} at 127.16.233.65:44551
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab.
Entry for principal kudu/127.16.233.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab.
I20260810 22:33:38.710194 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.66:0
--local_ip_for_outbound_sockets=127.16.233.66
--webserver_interface=127.16.233.66
--webserver_port=0
--tserver_master_addrs=127.16.233.126:39883
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab
--principal=kudu/127.16.233.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:41291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:33:38.883756 17592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:33:38.884079 17592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:33:38.884132 17592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:33:38.888646 17592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:33:38.888779 17592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:33:38.888926 17592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.66
I20260810 22:33:38.894987 17592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:41291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab
--principal=kudu/127.16.233.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.233.66
--webserver_port=0
--tserver_master_addrs=127.16.233.126:39883
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.66
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:33:38.896173 17592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:33:38.897336 17592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:33:38.908600 17598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:33:38.908656 17597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:33:38.908851 17592 server_base.cc:1034] running on GCE node
W20260810 22:33:38.908641 17600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:33:38.909361 17592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:33:38.910025 17592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:33:38.911234 17592 hybrid_clock.cc:648] HybridClock initialized: now 1786401218911200 us; error 45 us; skew 500 ppm
Aug 10 22:33:38 dist-test-slave-77v9 krb5kdc[17324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401218, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:33:38.914414 17592 init.cc:377] Logged in from keytab as kudu/127.16.233.66@KRBTEST.COM (short username kudu)
I20260810 22:33:38.915421 17592 webserver.cc:460] Webserver started at http://127.16.233.66:38617/ using document root <none> and password file <none>
I20260810 22:33:38.916028 17592 fs_manager.cc:359] Metadata directory not provided
I20260810 22:33:38.916133 17592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:33:38.916395 17592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:33:38.918813 17592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/minicluster-data/ts-1/data/instance:
uuid: "86b25a82aa174fe1acbe9e01107df6ac"
format_stamp: "Formatted at 2026-08-10 22:33:38 on dist-test-slave-77v9"
server_key: "2fee8dc5c90e293e4af6c09bd462d136"
server_key_iv: "d391db146208e9c14796d0b4208e1e7f"
server_key_version: "encryptionkey@0"
I20260810 22:33:38.919449 17592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/minicluster-data/ts-1/wal/instance:
uuid: "86b25a82aa174fe1acbe9e01107df6ac"
format_stamp: "Formatted at 2026-08-10 22:33:38 on dist-test-slave-77v9"
server_key: "2fee8dc5c90e293e4af6c09bd462d136"
server_key_iv: "d391db146208e9c14796d0b4208e1e7f"
server_key_version: "encryptionkey@0"
I20260810 22:33:38.924207 17592 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260810 22:33:38.927233 17609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:38.928601 17592 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 22:33:38.928892 17592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/minicluster-data/ts-1/wal
uuid: "86b25a82aa174fe1acbe9e01107df6ac"
format_stamp: "Formatted at 2026-08-10 22:33:38 on dist-test-slave-77v9"
server_key: "2fee8dc5c90e293e4af6c09bd462d136"
server_key_iv: "d391db146208e9c14796d0b4208e1e7f"
server_key_version: "encryptionkey@0"
I20260810 22:33:38.929032 17592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:33:38.951716 17592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:33:38.955540 17592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:33:38.955718 17592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:33:38.956354 17592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:33:38.957257 17592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:33:38.957306 17592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:38.957347 17592 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:33:38.957409 17592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:38.973536 17592 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.66:35351
I20260810 22:33:38.973709 17733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.66:35351 every 8 connection(s)
I20260810 22:33:38.974604 17592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
I20260810 22:33:38.980509 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 17592
I20260810 22:33:38.980612 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/minicluster-data/ts-1/wal/instance
I20260810 22:33:38.980940 17317 external_mini_cluster.cc:1428] Setting key 05c4a7efe324031460dceab1fe48fb1c
Aug 10 22:33:38 dist-test-slave-77v9 krb5kdc[17324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401218, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.66@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:33:38.992007 17418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:38.976795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:47049 (local address 127.16.233.126:39883)
0810 22:33:38.976986 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:38.976990 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:38.978258 (+ 1268us) server_negotiation.cc:407] Connection header received
0810 22:33:38.979332 (+ 1074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:38.979335 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:38.979386 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:38.979475 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:38.981161 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:38.981766 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:38.983887 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:38.984285 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:38.984413 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:38.988060 (+ 3647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:38.988063 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:38.988065 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:38.988093 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:38.989729 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:38.990331 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:38.990335 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:38.990337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:38.990434 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:38.990810 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:38.990814 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:38.990815 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:38.991022 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:38.991137 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:38.991688 (+ 551us) server_negotiation.cc:299] Negotiation successful
0810 22:33:38.991830 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260810 22:33:38.993024 17736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:38.977239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39883 (local address 127.16.233.66:47049)
0810 22:33:38.978098 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:38.978142 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:38.979113 (+ 971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:38.979642 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:38.979653 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:38.980169 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:33:38.980935 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:38.980952 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:38.981930 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:38.981934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:38.983738 (+ 1804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:38.983751 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:38.984455 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:38.984457 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:38.984651 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:38.985566 (+ 915us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:33:38.985600 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:33:38.987774 (+ 2174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:33:38.989875 (+ 2101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:38.989882 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:38.989901 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:38.990205 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:38.990534 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:38.990538 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:38.990540 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:38.990699 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:38.991136 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:33:38.991145 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:33:38.991426 (+ 281us) client_negotiation.cc:769] Sending connection context
0810 22:33:38.991625 (+ 199us) client_negotiation.cc:240] Negotiation successful
0810 22:33:38.992020 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":151,"threads_started":1}
WARNING: no policy specified for kudu/127.16.233.67@KRBTEST.COM; defaulting to no policy
I20260810 22:33:38.994318 17734 heartbeater.cc:347] Connected to a master server at 127.16.233.126:39883
I20260810 22:33:38.994614 17734 heartbeater.cc:464] Registering TS with master...
I20260810 22:33:38.995198 17734 heartbeater.cc:511] Master 127.16.233.126:39883 requested a full tablet report, sending...
I20260810 22:33:38.996387 17374 ts_manager.cc:194] Registered new tserver with Master: 86b25a82aa174fe1acbe9e01107df6ac (127.16.233.66:35351)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.67@KRBTEST.COM" created.
I20260810 22:33:38.997190 17374 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.66@KRBTEST.COM'} at 127.16.233.66:47049
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab.
Entry for principal kudu/127.16.233.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab.
I20260810 22:33:39.018285 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.67:0
--local_ip_for_outbound_sockets=127.16.233.67
--webserver_interface=127.16.233.67
--webserver_port=0
--tserver_master_addrs=127.16.233.126:39883
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab
--principal=kudu/127.16.233.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:41291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:33:39.175166 17739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:33:39.175539 17739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:33:39.175622 17739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:33:39.180239 17739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:33:39.180361 17739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:33:39.180461 17739 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.67
I20260810 22:33:39.186406 17739 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:41291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab
--principal=kudu/127.16.233.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.233.67
--webserver_port=0
--tserver_master_addrs=127.16.233.126:39883
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17739
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.67
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:33:39.187709 17739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:33:39.188797 17739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:33:39.200428 17748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:33:39.200428 17745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:33:39.200428 17746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:33:39.201103 17739 server_base.cc:1034] running on GCE node
I20260810 22:33:39.201557 17739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:33:39.202288 17739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:33:39.203465 17739 hybrid_clock.cc:648] HybridClock initialized: now 1786401219203435 us; error 47 us; skew 500 ppm
Aug 10 22:33:39 dist-test-slave-77v9 krb5kdc[17324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401219, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:33:39.207562 17739 init.cc:377] Logged in from keytab as kudu/127.16.233.67@KRBTEST.COM (short username kudu)
I20260810 22:33:39.208725 17739 webserver.cc:460] Webserver started at http://127.16.233.67:38275/ using document root <none> and password file <none>
I20260810 22:33:39.209371 17739 fs_manager.cc:359] Metadata directory not provided
I20260810 22:33:39.209473 17739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:33:39.209722 17739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:33:39.212180 17739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/minicluster-data/ts-2/data/instance:
uuid: "f7a2b9c86a8748448f5a2b14154926c4"
format_stamp: "Formatted at 2026-08-10 22:33:39 on dist-test-slave-77v9"
server_key: "000f519d4ffc9d981c9a8b9e159af7d1"
server_key_iv: "552079369be44e8ea25a588b4fb9ffc9"
server_key_version: "encryptionkey@0"
I20260810 22:33:39.212968 17739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/minicluster-data/ts-2/wal/instance:
uuid: "f7a2b9c86a8748448f5a2b14154926c4"
format_stamp: "Formatted at 2026-08-10 22:33:39 on dist-test-slave-77v9"
server_key: "000f519d4ffc9d981c9a8b9e159af7d1"
server_key_iv: "552079369be44e8ea25a588b4fb9ffc9"
server_key_version: "encryptionkey@0"
I20260810 22:33:39.217471 17739 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260810 22:33:39.220259 17755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:39.221786 17739 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 22:33:39.221900 17739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/minicluster-data/ts-2/wal
uuid: "f7a2b9c86a8748448f5a2b14154926c4"
format_stamp: "Formatted at 2026-08-10 22:33:39 on dist-test-slave-77v9"
server_key: "000f519d4ffc9d981c9a8b9e159af7d1"
server_key_iv: "552079369be44e8ea25a588b4fb9ffc9"
server_key_version: "encryptionkey@0"
I20260810 22:33:39.221995 17739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:33:39.244989 17739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:33:39.249361 17739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:33:39.249670 17739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:33:39.250615 17739 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:33:39.251658 17739 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:33:39.251751 17739 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:39.251824 17739 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:33:39.251885 17739 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:39.271266 17739 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.67:36407
I20260810 22:33:39.271390 17883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.67:36407 every 8 connection(s)
I20260810 22:33:39.272328 17739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
I20260810 22:33:39.278119 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 17739
I20260810 22:33:39.278280 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786401215402979-17317-0/minicluster-data/ts-2/wal/instance
I20260810 22:33:39.278632 17317 external_mini_cluster.cc:1428] Setting key 2a257bb765d6b7b236b0a1b43fb0ddfb
Aug 10 22:33:39 dist-test-slave-77v9 krb5kdc[17324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401219, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.67@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:33:39.289460 17418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:39.274554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:55961 (local address 127.16.233.126:39883)
0810 22:33:39.274719 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:39.274723 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:39.276411 (+ 1688us) server_negotiation.cc:407] Connection header received
0810 22:33:39.277690 (+ 1279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:39.277694 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:39.277756 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:39.277868 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:39.279829 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:39.280525 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:39.282171 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:39.282405 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:39.282524 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:39.285577 (+ 3053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:39.285581 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:39.285584 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:39.285617 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:39.287196 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:39.287745 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:39.287753 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:39.287755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:39.287835 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:39.288210 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:39.288213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:39.288215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:39.288426 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:39.288540 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:39.289124 (+ 584us) server_negotiation.cc:299] Negotiation successful
0810 22:33:39.289274 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 22:33:39.290402 17886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:39.274940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39883 (local address 127.16.233.67:55961)
0810 22:33:39.276227 (+ 1287us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:39.276273 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:39.277458 (+ 1185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:39.278043 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:39.278050 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:39.278732 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:33:39.279648 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:39.279677 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:39.280651 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:39.280654 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:39.282043 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:39.282053 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:39.282562 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:39.282566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:39.282808 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:39.283599 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:33:39.283616 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:33:39.285395 (+ 1779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:33:39.287331 (+ 1936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:39.287338 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:39.287353 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:39.287627 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:39.287943 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:39.287946 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:39.287949 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:39.288102 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:39.288538 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:33:39.288545 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:33:39.288802 (+ 257us) client_negotiation.cc:769] Sending connection context
0810 22:33:39.289071 (+ 269us) client_negotiation.cc:240] Negotiation successful
0810 22:33:39.289424 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1073,"thread_start_us":778,"threads_started":1}
I20260810 22:33:39.291754 17884 heartbeater.cc:347] Connected to a master server at 127.16.233.126:39883
I20260810 22:33:39.292038 17884 heartbeater.cc:464] Registering TS with master...
I20260810 22:33:39.292977 17884 heartbeater.cc:511] Master 127.16.233.126:39883 requested a full tablet report, sending...
I20260810 22:33:39.294312 17374 ts_manager.cc:194] Registered new tserver with Master: f7a2b9c86a8748448f5a2b14154926c4 (127.16.233.67:36407)
I20260810 22:33:39.294998 17374 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.67@KRBTEST.COM'} at 127.16.233.67:55961
I20260810 22:33:39.306366 17317 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 22:33:39 dist-test-slave-77v9 krb5kdc[17324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401219, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 22:33:39 dist-test-slave-77v9 krb5kdc[17324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401219, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.233.65@KRBTEST.COM
I20260810 22:33:39.339010 17891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:39.328978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50422 (local address 127.16.233.65:33211)
0810 22:33:39.329351 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:39.329357 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:39.329474 (+ 117us) server_negotiation.cc:407] Connection header received
0810 22:33:39.329608 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:39.329613 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:39.329810 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:39.329938 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:39.330653 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:39.331450 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:39.332284 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:39.332680 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:39.332766 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:39.335454 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:39.335464 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:39.335478 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:39.335505 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:39.337181 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:39.337603 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:39.337608 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:39.337610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:39.337697 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:39.337942 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:39.337944 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:39.337946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:39.338206 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:39.338386 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:39.338661 (+ 275us) server_negotiation.cc:299] Negotiation successful
0810 22:33:39.338798 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":141,"threads_started":1}
W20260810 22:33:39.340195 17490 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50422
Aug 10 22:33:39 dist-test-slave-77v9 krb5kdc[17324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 22:33:39 dist-test-slave-77v9 krb5kdc[17324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.65@KRBTEST.COM
I20260810 22:33:39.371590 17891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:39.362527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50438 (local address 127.16.233.65:33211)
0810 22:33:39.362716 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:39.362720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:39.362738 (+ 18us) server_negotiation.cc:407] Connection header received
0810 22:33:39.362904 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:39.362906 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:39.362954 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:39.363030 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:39.363898 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:39.364442 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:39.365436 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:39.365666 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:39.365757 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:39.368486 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:39.368489 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:39.368492 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:39.368521 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:39.370033 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:39.370464 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:39.370466 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:39.370468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:39.370544 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:39.370781 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:39.370783 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:39.370785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:39.370959 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:39.371065 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:39.371268 (+ 203us) server_negotiation.cc:299] Negotiation successful
0810 22:33:39.371376 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260810 22:33:39.372957 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 17445
I20260810 22:33:39.381423 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 17592
I20260810 22:33:39.392208 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 17739
I20260810 22:33:39.400063 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 17339
2026-08-10T22:33:39Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3958 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:33:39 dist-test-slave-77v9 krb5kdc[17899](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:33:39 dist-test-slave-77v9 krb5kdc[17899](info): set up 2 sockets
Aug 10 22:33:39 dist-test-slave-77v9 krb5kdc[17899](info): commencing operation
krb5kdc: starting...
W20260810 22:33:40.591539 17317 mini_kdc.cc:121] Time spent starting KDC: real 1.164s user 0.003s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Aug 10 22:33:40 dist-test-slave-77v9 krb5kdc[17899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:33:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:33:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.233.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab.
Entry for principal kudu/127.16.233.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab.
I20260810 22:33:40.741267 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:39507
--webserver_interface=127.16.233.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab
--principal=kudu/127.16.233.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:33573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.126:39507
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:33:40.858314 17915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:33:40.858690 17915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:33:40.858783 17915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:33:40.863080 17915 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:33:40.863220 17915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:33:40.863260 17915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:33:40.863302 17915 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:33:40.863339 17915 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:33:40.869234 17915 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:33573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.126:39507
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab
--principal=kudu/127.16.233.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:39507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17915
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:33:40.870481 17915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:33:40.871593 17915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:33:40.880780 17925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:33:40.880910 17922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:33:40.880896 17923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:33:40.881150 17915 server_base.cc:1034] running on GCE node
I20260810 22:33:40.881888 17915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:33:40.883383 17915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:33:40.884975 17915 hybrid_clock.cc:648] HybridClock initialized: now 1786401220884983 us; error 115 us; skew 500 ppm
Aug 10 22:33:40 dist-test-slave-77v9 krb5kdc[17899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401220, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:33:40.892885 17915 init.cc:377] Logged in from keytab as kudu/127.16.233.126@KRBTEST.COM (short username kudu)
I20260810 22:33:40.894027 17915 webserver.cc:460] Webserver started at http://127.16.233.126:36615/ using document root <none> and password file <none>
I20260810 22:33:40.894693 17915 fs_manager.cc:359] Metadata directory not provided
I20260810 22:33:40.894805 17915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:33:40.895069 17915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:33:40.897393 17915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/minicluster-data/master-0/data/instance:
uuid: "d6120f210be44b0a8a95e0a6f1781b07"
format_stamp: "Formatted at 2026-08-10 22:33:40 on dist-test-slave-77v9"
server_key: "2816e33114495cc8903873a9153eff36"
server_key_iv: "1c87cede32257a47b4e117f25bf63c6a"
server_key_version: "encryptionkey@0"
I20260810 22:33:40.898025 17915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/minicluster-data/master-0/wal/instance:
uuid: "d6120f210be44b0a8a95e0a6f1781b07"
format_stamp: "Formatted at 2026-08-10 22:33:40 on dist-test-slave-77v9"
server_key: "2816e33114495cc8903873a9153eff36"
server_key_iv: "1c87cede32257a47b4e117f25bf63c6a"
server_key_version: "encryptionkey@0"
I20260810 22:33:40.902141 17915 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 22:33:40.904981 17932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:40.906471 17915 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:33:40.906667 17915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/minicluster-data/master-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/minicluster-data/master-0/wal
uuid: "d6120f210be44b0a8a95e0a6f1781b07"
format_stamp: "Formatted at 2026-08-10 22:33:40 on dist-test-slave-77v9"
server_key: "2816e33114495cc8903873a9153eff36"
server_key_iv: "1c87cede32257a47b4e117f25bf63c6a"
server_key_version: "encryptionkey@0"
I20260810 22:33:40.906826 17915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:33:40.918481 17915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:33:40.922284 17915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:33:40.922583 17915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:33:40.932093 17915 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.126:39507
I20260810 22:33:40.932129 17992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.126:39507 every 8 connection(s)
I20260810 22:33:40.933391 17915 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
I20260810 22:33:40.936657 17993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:33:40.939961 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 17915
I20260810 22:33:40.940099 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/minicluster-data/master-0/wal/instance
I20260810 22:33:40.940487 17317 external_mini_cluster.cc:1428] Setting key 023cc91b3e6376e2ba1259833f14d51c
I20260810 22:33:40.943807 17993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6120f210be44b0a8a95e0a6f1781b07: Bootstrap starting.
I20260810 22:33:40.946873 17993 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6120f210be44b0a8a95e0a6f1781b07: Neither blocks nor log segments found. Creating new log.
I20260810 22:33:40.948026 17993 log.cc:824] T 00000000000000000000000000000000 P d6120f210be44b0a8a95e0a6f1781b07: Log is configured to *not* fsync() on all Append() calls
I20260810 22:33:40.950978 17993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6120f210be44b0a8a95e0a6f1781b07: No bootstrap required, opened a new log
Aug 10 22:33:40 dist-test-slave-77v9 krb5kdc[17899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:33:40.956080 17993 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6120f210be44b0a8a95e0a6f1781b07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6120f210be44b0a8a95e0a6f1781b07" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39507 } }
I20260810 22:33:40.956418 17993 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6120f210be44b0a8a95e0a6f1781b07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:33:40.956506 17993 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6120f210be44b0a8a95e0a6f1781b07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6120f210be44b0a8a95e0a6f1781b07, State: Initialized, Role: FOLLOWER
I20260810 22:33:40.957124 17993 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6120f210be44b0a8a95e0a6f1781b07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6120f210be44b0a8a95e0a6f1781b07" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39507 } }
I20260810 22:33:40.957286 17993 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6120f210be44b0a8a95e0a6f1781b07 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:33:40.957386 17993 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6120f210be44b0a8a95e0a6f1781b07 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:33:40.957500 17993 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6120f210be44b0a8a95e0a6f1781b07 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:33:40.958750 17993 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6120f210be44b0a8a95e0a6f1781b07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6120f210be44b0a8a95e0a6f1781b07" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39507 } }
I20260810 22:33:40.959177 17993 leader_election.cc:302] T 00000000000000000000000000000000 P d6120f210be44b0a8a95e0a6f1781b07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6120f210be44b0a8a95e0a6f1781b07; no voters:
I20260810 22:33:40.959462 17996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:40.942407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33840 (local address 127.16.233.126:39507)
0810 22:33:40.942990 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:40.943010 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:40.943052 (+ 42us) server_negotiation.cc:407] Connection header received
0810 22:33:40.943786 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:40.943805 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:40.944247 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:40.944569 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:40.945788 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:40.946998 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:40.948439 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:40.949233 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:40.949376 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:40.952466 (+ 3090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:40.952473 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:40.952493 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:40.952531 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:40.955962 (+ 3431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:40.956475 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:40.956482 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:40.956494 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:40.956679 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:40.957033 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:40.957037 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:40.957039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:40.957646 (+ 607us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:40.957879 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:40.958189 (+ 310us) server_negotiation.cc:299] Negotiation successful
0810 22:33:40.958512 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":216,"threads_started":1}
I20260810 22:33:40.959592 17993 leader_election.cc:288] T 00000000000000000000000000000000 P d6120f210be44b0a8a95e0a6f1781b07 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:33:40.959687 17998 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6120f210be44b0a8a95e0a6f1781b07 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:33:40.959954 17998 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6120f210be44b0a8a95e0a6f1781b07 [term 1 LEADER]: Becoming Leader. State: Replica: d6120f210be44b0a8a95e0a6f1781b07, State: Running, Role: LEADER
I20260810 22:33:40.960394 17998 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6120f210be44b0a8a95e0a6f1781b07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6120f210be44b0a8a95e0a6f1781b07" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39507 } }
I20260810 22:33:40.961182 17993 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6120f210be44b0a8a95e0a6f1781b07 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:33:40.962481 18000 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6120f210be44b0a8a95e0a6f1781b07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6120f210be44b0a8a95e0a6f1781b07. Latest consensus state: current_term: 1 leader_uuid: "d6120f210be44b0a8a95e0a6f1781b07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6120f210be44b0a8a95e0a6f1781b07" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39507 } } }
I20260810 22:33:40.962618 18000 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6120f210be44b0a8a95e0a6f1781b07 [sys.catalog]: This master's current role is: LEADER
I20260810 22:33:40.963032 17999 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6120f210be44b0a8a95e0a6f1781b07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6120f210be44b0a8a95e0a6f1781b07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6120f210be44b0a8a95e0a6f1781b07" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39507 } } }
I20260810 22:33:40.963791 18009 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:33:40.966603 18009 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:33:40.968307 17999 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6120f210be44b0a8a95e0a6f1781b07 [sys.catalog]: This master's current role is: LEADER
I20260810 22:33:40.972218 18009 catalog_manager.cc:1347] Generated new cluster ID: ccbe69bda7ee400cbc2f5b803dabc25e
I20260810 22:33:40.972306 18009 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:33:40.982088 18009 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:33:40.983065 18009 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:33:40.989522 18009 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6120f210be44b0a8a95e0a6f1781b07: Generated new TSK 0
I20260810 22:33:40.990270 18009 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.233.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab.
Entry for principal kudu/127.16.233.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab.
I20260810 22:33:41.029382 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.65:0
--local_ip_for_outbound_sockets=127.16.233.65
--webserver_interface=127.16.233.65
--webserver_port=0
--tserver_master_addrs=127.16.233.126:39507
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab
--principal=kudu/127.16.233.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:33573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:33:41.148169 18024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:33:41.148588 18024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:33:41.148693 18024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:33:41.153002 18024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:33:41.153072 18024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:33:41.153175 18024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.65
I20260810 22:33:41.159185 18024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:33573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab
--principal=kudu/127.16.233.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.233.65
--webserver_port=0
--tserver_master_addrs=127.16.233.126:39507
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.65
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:33:41.160494 18024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:33:41.161639 18024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:33:41.172096 18029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:33:41.172290 18024 server_base.cc:1034] running on GCE node
W20260810 22:33:41.172101 18031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:33:41.172119 18033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:33:41.172966 18024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:33:41.173648 18024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:33:41.174883 18024 hybrid_clock.cc:648] HybridClock initialized: now 1786401221174851 us; error 49 us; skew 500 ppm
Aug 10 22:33:41 dist-test-slave-77v9 krb5kdc[17899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401221, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:33:41.178341 18024 init.cc:377] Logged in from keytab as kudu/127.16.233.65@KRBTEST.COM (short username kudu)
I20260810 22:33:41.179394 18024 webserver.cc:460] Webserver started at http://127.16.233.65:39245/ using document root <none> and password file <none>
I20260810 22:33:41.180110 18024 fs_manager.cc:359] Metadata directory not provided
I20260810 22:33:41.180222 18024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:33:41.180482 18024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:33:41.182837 18024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/minicluster-data/ts-0/data/instance:
uuid: "c77ac95af6784e648571601102962e01"
format_stamp: "Formatted at 2026-08-10 22:33:41 on dist-test-slave-77v9"
server_key: "9160cbc0ce2f98de87a9ff1891274e07"
server_key_iv: "fc899eea996df5501ce03fe1bd2f5885"
server_key_version: "encryptionkey@0"
I20260810 22:33:41.183476 18024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/minicluster-data/ts-0/wal/instance:
uuid: "c77ac95af6784e648571601102962e01"
format_stamp: "Formatted at 2026-08-10 22:33:41 on dist-test-slave-77v9"
server_key: "9160cbc0ce2f98de87a9ff1891274e07"
server_key_iv: "fc899eea996df5501ce03fe1bd2f5885"
server_key_version: "encryptionkey@0"
I20260810 22:33:41.187985 18024 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 22:33:41.190824 18042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:41.192253 18024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 22:33:41.192415 18024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/minicluster-data/ts-0/wal
uuid: "c77ac95af6784e648571601102962e01"
format_stamp: "Formatted at 2026-08-10 22:33:41 on dist-test-slave-77v9"
server_key: "9160cbc0ce2f98de87a9ff1891274e07"
server_key_iv: "fc899eea996df5501ce03fe1bd2f5885"
server_key_version: "encryptionkey@0"
I20260810 22:33:41.192550 18024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:33:41.218096 18024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:33:41.221804 18024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:33:41.222040 18024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:33:41.222694 18024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:33:41.223623 18024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:33:41.223697 18024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:41.223780 18024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:33:41.223814 18024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:41.234021 18024 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.65:37351
I20260810 22:33:41.234052 18164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.65:37351 every 8 connection(s)
I20260810 22:33:41.235010 18024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
I20260810 22:33:41.237833 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 18024
I20260810 22:33:41.237987 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/minicluster-data/ts-0/wal/instance
I20260810 22:33:41.238278 17317 external_mini_cluster.cc:1428] Setting key bb4ae1eae405b2f4ad83d532bb0d642d
Aug 10 22:33:41 dist-test-slave-77v9 krb5kdc[17899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401221, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.65@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.233.66@KRBTEST.COM; defaulting to no policy
I20260810 22:33:41.251693 17996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:41.237086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:57189 (local address 127.16.233.126:39507)
0810 22:33:41.237271 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:41.237286 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:41.238433 (+ 1147us) server_negotiation.cc:407] Connection header received
0810 22:33:41.239375 (+ 942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:41.239378 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:41.239432 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:41.239509 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:41.240749 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:41.241384 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:41.243093 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:41.243346 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:41.243443 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:41.246851 (+ 3408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:41.246855 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:41.246857 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:41.246891 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:41.248664 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:41.249416 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:41.249420 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:41.249422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:41.249536 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:41.250148 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:41.250152 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:41.250153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:41.250413 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:41.250588 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:41.251420 (+ 832us) server_negotiation.cc:299] Negotiation successful
0810 22:33:41.251557 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.66@KRBTEST.COM" created.
I20260810 22:33:41.252849 18167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:41.237446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39507 (local address 127.16.233.65:57189)
0810 22:33:41.238264 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:41.238324 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:41.239157 (+ 833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:41.239680 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:41.239687 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:41.240076 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:33:41.240587 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:41.240602 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:41.241536 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:41.241540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:41.242933 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:41.242948 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:41.243528 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:41.243535 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:41.243772 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:41.244508 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:33:41.244534 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:33:41.246609 (+ 2075us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:33:41.248860 (+ 2251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:41.248870 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:41.248890 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:41.249232 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:41.249666 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:41.249669 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:41.249670 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:41.249948 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:41.250716 (+ 768us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:33:41.250727 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:33:41.251122 (+ 395us) client_negotiation.cc:769] Sending connection context
0810 22:33:41.251317 (+ 195us) client_negotiation.cc:240] Negotiation successful
0810 22:33:41.251632 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":149,"threads_started":1}
I20260810 22:33:41.254398 18165 heartbeater.cc:347] Connected to a master server at 127.16.233.126:39507
I20260810 22:33:41.254667 18165 heartbeater.cc:464] Registering TS with master...
I20260810 22:33:41.255281 18165 heartbeater.cc:511] Master 127.16.233.126:39507 requested a full tablet report, sending...
I20260810 22:33:41.256870 17952 ts_manager.cc:194] Registered new tserver with Master: c77ac95af6784e648571601102962e01 (127.16.233.65:37351)
I20260810 22:33:41.258150 17952 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.65@KRBTEST.COM'} at 127.16.233.65:57189
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab.
Entry for principal kudu/127.16.233.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab.
I20260810 22:33:41.274322 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.66:0
--local_ip_for_outbound_sockets=127.16.233.66
--webserver_interface=127.16.233.66
--webserver_port=0
--tserver_master_addrs=127.16.233.126:39507
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab
--principal=kudu/127.16.233.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:33573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:33:41.391047 18170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:33:41.391451 18170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:33:41.391551 18170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:33:41.395998 18170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:33:41.396099 18170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:33:41.396195 18170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.66
I20260810 22:33:41.402201 18170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:33573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab
--principal=kudu/127.16.233.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.233.66
--webserver_port=0
--tserver_master_addrs=127.16.233.126:39507
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.66
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:33:41.403386 18170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:33:41.404500 18170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:33:41.414855 18175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:33:41.415100 18170 server_base.cc:1034] running on GCE node
W20260810 22:33:41.414897 18178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:33:41.414945 18176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:33:41.415674 18170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:33:41.416425 18170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:33:41.417636 18170 hybrid_clock.cc:648] HybridClock initialized: now 1786401221417560 us; error 89 us; skew 500 ppm
Aug 10 22:33:41 dist-test-slave-77v9 krb5kdc[17899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401221, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:33:41.421166 18170 init.cc:377] Logged in from keytab as kudu/127.16.233.66@KRBTEST.COM (short username kudu)
I20260810 22:33:41.422300 18170 webserver.cc:460] Webserver started at http://127.16.233.66:42999/ using document root <none> and password file <none>
I20260810 22:33:41.422936 18170 fs_manager.cc:359] Metadata directory not provided
I20260810 22:33:41.423044 18170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:33:41.423316 18170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:33:41.425801 18170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/minicluster-data/ts-1/data/instance:
uuid: "de84eb073d43405c9573a6db052549a0"
format_stamp: "Formatted at 2026-08-10 22:33:41 on dist-test-slave-77v9"
server_key: "610d773fa8b2fc80366c57c9520bcd63"
server_key_iv: "b44e4e1b603d968a3c75dbf3598c01ab"
server_key_version: "encryptionkey@0"
I20260810 22:33:41.426596 18170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/minicluster-data/ts-1/wal/instance:
uuid: "de84eb073d43405c9573a6db052549a0"
format_stamp: "Formatted at 2026-08-10 22:33:41 on dist-test-slave-77v9"
server_key: "610d773fa8b2fc80366c57c9520bcd63"
server_key_iv: "b44e4e1b603d968a3c75dbf3598c01ab"
server_key_version: "encryptionkey@0"
I20260810 22:33:41.436996 18170 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.007s sys 0.000s
I20260810 22:33:41.440459 18185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:41.442116 18170 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:33:41.442238 18170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/minicluster-data/ts-1/wal
uuid: "de84eb073d43405c9573a6db052549a0"
format_stamp: "Formatted at 2026-08-10 22:33:41 on dist-test-slave-77v9"
server_key: "610d773fa8b2fc80366c57c9520bcd63"
server_key_iv: "b44e4e1b603d968a3c75dbf3598c01ab"
server_key_version: "encryptionkey@0"
I20260810 22:33:41.442345 18170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:33:41.457605 18170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:33:41.461304 18170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:33:41.461500 18170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:33:41.462193 18170 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:33:41.463109 18170 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:33:41.463158 18170 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:41.463200 18170 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:33:41.463215 18170 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:41.472591 18170 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.66:42399
I20260810 22:33:41.472688 18308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.66:42399 every 8 connection(s)
I20260810 22:33:41.473665 18170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
I20260810 22:33:41.482980 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 18170
I20260810 22:33:41.483223 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/minicluster-data/ts-1/wal/instance
I20260810 22:33:41.483532 17317 external_mini_cluster.cc:1428] Setting key 4b275d158298d6aa1c467de37821e749
Aug 10 22:33:41 dist-test-slave-77v9 krb5kdc[17899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401221, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.66@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:33:41.489161 17996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:41.475664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:53973 (local address 127.16.233.126:39507)
0810 22:33:41.475920 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:41.475925 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:41.476777 (+ 852us) server_negotiation.cc:407] Connection header received
0810 22:33:41.478026 (+ 1249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:41.478029 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:41.478080 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:41.478161 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:41.479341 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:41.479931 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:41.481317 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:41.481537 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:41.481617 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:41.484472 (+ 2855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:41.484476 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:41.484479 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:41.484504 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:41.486498 (+ 1994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:41.487135 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:41.487141 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:41.487146 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:41.487248 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:41.487790 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:41.487792 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:41.487794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:41.488000 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:41.488081 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:41.488892 (+ 811us) server_negotiation.cc:299] Negotiation successful
0810 22:33:41.489029 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 22:33:41.490478 18312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:41.476084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39507 (local address 127.16.233.66:53973)
0810 22:33:41.476602 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:41.476649 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:41.477814 (+ 1165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:41.478339 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:41.478345 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:41.478714 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:33:41.479181 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:41.479194 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:41.480080 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:41.480083 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:41.481173 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:41.481184 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:41.481678 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:41.481681 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:41.481852 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:41.482466 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:33:41.482484 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:33:41.484320 (+ 1836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:33:41.486654 (+ 2334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:41.486659 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:41.486674 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:41.487024 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:41.487353 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:41.487355 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:41.487356 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:41.487674 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:41.488115 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:33:41.488123 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:33:41.488504 (+ 381us) client_negotiation.cc:769] Sending connection context
0810 22:33:41.488781 (+ 277us) client_negotiation.cc:240] Negotiation successful
0810 22:33:41.489226 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":128,"threads_started":1}
I20260810 22:33:41.491860 18309 heartbeater.cc:347] Connected to a master server at 127.16.233.126:39507
I20260810 22:33:41.492154 18309 heartbeater.cc:464] Registering TS with master...
I20260810 22:33:41.492744 18309 heartbeater.cc:511] Master 127.16.233.126:39507 requested a full tablet report, sending...
I20260810 22:33:41.494120 17952 ts_manager.cc:194] Registered new tserver with Master: de84eb073d43405c9573a6db052549a0 (127.16.233.66:42399)
I20260810 22:33:41.494793 17952 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.66@KRBTEST.COM'} at 127.16.233.66:53973
WARNING: no policy specified for kudu/127.16.233.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab.
Entry for principal kudu/127.16.233.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab.
I20260810 22:33:41.517357 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.67:0
--local_ip_for_outbound_sockets=127.16.233.67
--webserver_interface=127.16.233.67
--webserver_port=0
--tserver_master_addrs=127.16.233.126:39507
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab
--principal=kudu/127.16.233.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:33573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:33:41.642484 18316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:33:41.642776 18316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:33:41.642823 18316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:33:41.647346 18316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:33:41.647426 18316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:33:41.647521 18316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.67
I20260810 22:33:41.653766 18316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:33573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab
--principal=kudu/127.16.233.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.233.67
--webserver_port=0
--tserver_master_addrs=127.16.233.126:39507
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.67
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:33:41.655200 18316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:33:41.656301 18316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:33:41.667919 18321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:33:41.667897 18322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:33:41.668277 18316 server_base.cc:1034] running on GCE node
W20260810 22:33:41.667920 18325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:33:41.668915 18316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:33:41.669602 18316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:33:41.670869 18316 hybrid_clock.cc:648] HybridClock initialized: now 1786401221670845 us; error 45 us; skew 500 ppm
Aug 10 22:33:41 dist-test-slave-77v9 krb5kdc[17899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401221, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:33:41.673956 18316 init.cc:377] Logged in from keytab as kudu/127.16.233.67@KRBTEST.COM (short username kudu)
I20260810 22:33:41.674932 18316 webserver.cc:460] Webserver started at http://127.16.233.67:34585/ using document root <none> and password file <none>
I20260810 22:33:41.675628 18316 fs_manager.cc:359] Metadata directory not provided
I20260810 22:33:41.675697 18316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:33:41.675889 18316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:33:41.678262 18316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/minicluster-data/ts-2/data/instance:
uuid: "5f8ee3a3bfa8400f8751c64e4e48a6f2"
format_stamp: "Formatted at 2026-08-10 22:33:41 on dist-test-slave-77v9"
server_key: "c998cbc4dc971a6bfc19b3d65608a36a"
server_key_iv: "15390ccc1c9c4bc23289612beb735c1b"
server_key_version: "encryptionkey@0"
I20260810 22:33:41.678843 18316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/minicluster-data/ts-2/wal/instance:
uuid: "5f8ee3a3bfa8400f8751c64e4e48a6f2"
format_stamp: "Formatted at 2026-08-10 22:33:41 on dist-test-slave-77v9"
server_key: "c998cbc4dc971a6bfc19b3d65608a36a"
server_key_iv: "15390ccc1c9c4bc23289612beb735c1b"
server_key_version: "encryptionkey@0"
I20260810 22:33:41.683101 18316 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260810 22:33:41.686123 18332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:41.687556 18316 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 22:33:41.687668 18316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/minicluster-data/ts-2/wal
uuid: "5f8ee3a3bfa8400f8751c64e4e48a6f2"
format_stamp: "Formatted at 2026-08-10 22:33:41 on dist-test-slave-77v9"
server_key: "c998cbc4dc971a6bfc19b3d65608a36a"
server_key_iv: "15390ccc1c9c4bc23289612beb735c1b"
server_key_version: "encryptionkey@0"
I20260810 22:33:41.687990 18316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:33:41.706557 18316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:33:41.710246 18316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:33:41.710493 18316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:33:41.711239 18316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:33:41.712215 18316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:33:41.712275 18316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:41.712344 18316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:33:41.712381 18316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:41.722561 18316 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.67:36247
I20260810 22:33:41.722654 18455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.67:36247 every 8 connection(s)
I20260810 22:33:41.723618 18316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
I20260810 22:33:41.726702 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 18316
I20260810 22:33:41.726867 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786401215402979-17317-0/minicluster-data/ts-2/wal/instance
I20260810 22:33:41.727221 17317 external_mini_cluster.cc:1428] Setting key e3b2e1eef6bd3041d63399fc7c228940
Aug 10 22:33:41 dist-test-slave-77v9 krb5kdc[17899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401221, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.67@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:33:41.739333 17996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:41.725762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:33505 (local address 127.16.233.126:39507)
0810 22:33:41.726083 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:41.726088 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:41.727411 (+ 1323us) server_negotiation.cc:407] Connection header received
0810 22:33:41.728797 (+ 1386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:41.728799 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:41.728898 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:41.728976 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:41.730357 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:41.730887 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:41.732242 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:41.732475 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:41.732555 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:41.735426 (+ 2871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:41.735430 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:41.735432 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:41.735459 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:41.737124 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:41.737671 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:41.737673 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:41.737675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:41.737753 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:41.738141 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:41.738143 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:41.738144 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:41.738325 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:41.738408 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:41.739085 (+ 677us) server_negotiation.cc:299] Negotiation successful
0810 22:33:41.739193 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260810 22:33:41.740654 18458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:41.726222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39507 (local address 127.16.233.67:33505)
0810 22:33:41.727146 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:41.727249 (+ 103us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:41.728533 (+ 1284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:41.729169 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:41.729178 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:41.729683 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:33:41.730216 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:41.730231 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:41.731038 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:41.731041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:41.732117 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:41.732127 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:41.732619 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:41.732622 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:41.732777 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:41.733495 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:33:41.733513 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:33:41.735228 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:33:41.737239 (+ 2011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:41.737245 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:41.737259 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:41.737574 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:41.737872 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:41.737875 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:41.737877 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:41.738048 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:41.738473 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:33:41.738478 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:33:41.738831 (+ 353us) client_negotiation.cc:769] Sending connection context
0810 22:33:41.739030 (+ 199us) client_negotiation.cc:240] Negotiation successful
0810 22:33:41.739335 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":176,"threads_started":1}
I20260810 22:33:41.742159 18456 heartbeater.cc:347] Connected to a master server at 127.16.233.126:39507
I20260810 22:33:41.742441 18456 heartbeater.cc:464] Registering TS with master...
I20260810 22:33:41.742980 18456 heartbeater.cc:511] Master 127.16.233.126:39507 requested a full tablet report, sending...
I20260810 22:33:41.744257 17952 ts_manager.cc:194] Registered new tserver with Master: 5f8ee3a3bfa8400f8751c64e4e48a6f2 (127.16.233.67:36247)
I20260810 22:33:41.744997 17952 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.67@KRBTEST.COM'} at 127.16.233.67:33505
I20260810 22:33:41.754343 17317 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 22:33:41.768373 17996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:41.759523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33856 (local address 127.16.233.126:39507)
0810 22:33:41.759694 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:41.759698 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:41.759920 (+ 222us) server_negotiation.cc:407] Connection header received
0810 22:33:41.760043 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:41.760046 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:41.760100 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:41.760184 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:41.761238 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:41.761821 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:41.762909 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:41.763127 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:41.763213 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:41.764725 (+ 1512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:41.764728 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:41.764730 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:41.764756 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:41.766423 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:41.766942 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:41.766945 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:41.766946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:41.767026 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:41.767422 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:41.767425 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:41.767426 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:41.767671 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:41.767858 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:41.768131 (+ 273us) server_negotiation.cc:299] Negotiation successful
0810 22:33:41.768238 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260810 22:33:41.775135 17952 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33856:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 22:33:41.778945 17952 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:33:41.797633 18468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:41.791186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.65:37351 (local address 127.0.0.1:56294)
0810 22:33:41.792070 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:41.792129 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:41.792261 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:41.793080 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:41.793086 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:41.793117 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:33:41.793226 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:41.793231 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:41.794385 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:41.794391 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:41.795671 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:41.795682 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:41.796991 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:41.796995 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:41.797118 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:41.797169 (+ 51us) client_negotiation.cc:769] Sending connection context
0810 22:33:41.797292 (+ 123us) client_negotiation.cc:240] Negotiation successful
0810 22:33:41.797447 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":695,"thread_start_us":142,"threads_started":1}
I20260810 22:33:41.797989 18469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:41.791372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56294 (local address 127.16.233.65:37351)
0810 22:33:41.792261 (+ 889us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:41.792268 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:41.792294 (+ 26us) server_negotiation.cc:407] Connection header received
0810 22:33:41.792486 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:41.792494 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:41.792712 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:41.792886 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:33:41.793367 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:41.794251 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:41.795836 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:41.796727 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:41.796895 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:41.797154 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:41.797731 (+ 577us) server_negotiation.cc:299] Negotiation successful
0810 22:33:41.797834 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":789,"thread_start_us":102,"threads_started":1}
I20260810 22:33:41.797633 18471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:41.791960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.66:42399 (local address 127.0.0.1:58768)
0810 22:33:41.792750 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:41.792763 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:41.792992 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:41.793456 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:41.793459 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:41.793481 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:33:41.793554 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:41.793560 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:41.794910 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:41.794913 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:41.796064 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:41.796073 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:41.797213 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:41.797218 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:41.797341 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:41.797363 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 22:33:41.797421 (+ 58us) client_negotiation.cc:240] Negotiation successful
0810 22:33:41.797475 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":734,"thread_start_us":86,"threads_started":1}
I20260810 22:33:41.798120 18472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:41.792174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58768 (local address 127.16.233.66:42399)
0810 22:33:41.792617 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:41.792630 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:41.792921 (+ 291us) server_negotiation.cc:407] Connection header received
0810 22:33:41.793143 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:41.793148 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:41.793313 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:41.793464 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:33:41.793646 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:41.794756 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:41.796187 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:41.796830 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:41.796996 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:41.797735 (+ 739us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:41.797791 (+ 56us) server_negotiation.cc:299] Negotiation successful
0810 22:33:41.797894 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":117,"threads_started":1}
I20260810 22:33:41.798552 18470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:41.791583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60522 (local address 127.16.233.67:36247)
0810 22:33:41.792065 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:41.792078 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:41.792613 (+ 535us) server_negotiation.cc:407] Connection header received
0810 22:33:41.792700 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:41.792705 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:41.792945 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:41.793091 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:33:41.793537 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:41.794749 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:41.796305 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:41.797201 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:41.797351 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:41.797489 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:41.797598 (+ 109us) server_negotiation.cc:299] Negotiation successful
0810 22:33:41.797701 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":179,"threads_started":1}
I20260810 22:33:41.799178 18467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:41.791199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.67:36247 (local address 127.0.0.1:60522)
0810 22:33:41.792395 (+ 1196us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:41.792410 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:41.792510 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:41.793249 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:41.793252 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:41.793269 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:33:41.793403 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:41.793412 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:41.794941 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:41.794944 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:41.796174 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:41.796184 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:41.797351 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:41.797355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:41.797455 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:41.797477 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 22:33:41.797565 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 22:33:41.797645 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1088,"thread_start_us":109,"threads_started":1}
I20260810 22:33:41.801412 18239 tablet_service.cc:1467] Processing CreateTablet for tablet 7462d9f327334621872d32bcbbdebd0a (DEFAULT_TABLE table=test-table [id=53a3c8779785480fa3aced1d4a5eeca3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:33:41.801893 18387 tablet_service.cc:1467] Processing CreateTablet for tablet 7462d9f327334621872d32bcbbdebd0a (DEFAULT_TABLE table=test-table [id=53a3c8779785480fa3aced1d4a5eeca3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:33:41.802667 18239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7462d9f327334621872d32bcbbdebd0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:33:41.802932 18387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7462d9f327334621872d32bcbbdebd0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:33:41.802524 18096 tablet_service.cc:1467] Processing CreateTablet for tablet 7462d9f327334621872d32bcbbdebd0a (DEFAULT_TABLE table=test-table [id=53a3c8779785480fa3aced1d4a5eeca3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:33:41.803484 18096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7462d9f327334621872d32bcbbdebd0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:33:41.810425 18473 tablet_bootstrap.cc:492] T 7462d9f327334621872d32bcbbdebd0a P 5f8ee3a3bfa8400f8751c64e4e48a6f2: Bootstrap starting.
I20260810 22:33:41.812211 18474 tablet_bootstrap.cc:492] T 7462d9f327334621872d32bcbbdebd0a P c77ac95af6784e648571601102962e01: Bootstrap starting.
I20260810 22:33:41.813385 18475 tablet_bootstrap.cc:492] T 7462d9f327334621872d32bcbbdebd0a P de84eb073d43405c9573a6db052549a0: Bootstrap starting.
I20260810 22:33:41.814875 18474 tablet_bootstrap.cc:654] T 7462d9f327334621872d32bcbbdebd0a P c77ac95af6784e648571601102962e01: Neither blocks nor log segments found. Creating new log.
I20260810 22:33:41.815186 18473 tablet_bootstrap.cc:654] T 7462d9f327334621872d32bcbbdebd0a P 5f8ee3a3bfa8400f8751c64e4e48a6f2: Neither blocks nor log segments found. Creating new log.
I20260810 22:33:41.815949 18474 log.cc:824] T 7462d9f327334621872d32bcbbdebd0a P c77ac95af6784e648571601102962e01: Log is configured to *not* fsync() on all Append() calls
I20260810 22:33:41.816988 18475 tablet_bootstrap.cc:654] T 7462d9f327334621872d32bcbbdebd0a P de84eb073d43405c9573a6db052549a0: Neither blocks nor log segments found. Creating new log.
I20260810 22:33:41.817286 18473 log.cc:824] T 7462d9f327334621872d32bcbbdebd0a P 5f8ee3a3bfa8400f8751c64e4e48a6f2: Log is configured to *not* fsync() on all Append() calls
I20260810 22:33:41.818238 18475 log.cc:824] T 7462d9f327334621872d32bcbbdebd0a P de84eb073d43405c9573a6db052549a0: Log is configured to *not* fsync() on all Append() calls
I20260810 22:33:41.818281 18474 tablet_bootstrap.cc:492] T 7462d9f327334621872d32bcbbdebd0a P c77ac95af6784e648571601102962e01: No bootstrap required, opened a new log
I20260810 22:33:41.818529 18474 ts_tablet_manager.cc:1397] T 7462d9f327334621872d32bcbbdebd0a P c77ac95af6784e648571601102962e01: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 22:33:41.819623 18473 tablet_bootstrap.cc:492] T 7462d9f327334621872d32bcbbdebd0a P 5f8ee3a3bfa8400f8751c64e4e48a6f2: No bootstrap required, opened a new log
I20260810 22:33:41.819811 18473 ts_tablet_manager.cc:1397] T 7462d9f327334621872d32bcbbdebd0a P 5f8ee3a3bfa8400f8751c64e4e48a6f2: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260810 22:33:41.820529 18475 tablet_bootstrap.cc:492] T 7462d9f327334621872d32bcbbdebd0a P de84eb073d43405c9573a6db052549a0: No bootstrap required, opened a new log
I20260810 22:33:41.820708 18475 ts_tablet_manager.cc:1397] T 7462d9f327334621872d32bcbbdebd0a P de84eb073d43405c9573a6db052549a0: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260810 22:33:41.823475 18474 raft_consensus.cc:348] T 7462d9f327334621872d32bcbbdebd0a P c77ac95af6784e648571601102962e01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77ac95af6784e648571601102962e01" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 37351 } } peers { permanent_uuid: "5f8ee3a3bfa8400f8751c64e4e48a6f2" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 36247 } } peers { permanent_uuid: "de84eb073d43405c9573a6db052549a0" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 42399 } }
I20260810 22:33:41.823733 18474 raft_consensus.cc:374] T 7462d9f327334621872d32bcbbdebd0a P c77ac95af6784e648571601102962e01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:33:41.823789 18474 raft_consensus.cc:729] T 7462d9f327334621872d32bcbbdebd0a P c77ac95af6784e648571601102962e01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c77ac95af6784e648571601102962e01, State: Initialized, Role: FOLLOWER
I20260810 22:33:41.824229 18474 consensus_queue.cc:260] T 7462d9f327334621872d32bcbbdebd0a P c77ac95af6784e648571601102962e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77ac95af6784e648571601102962e01" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 37351 } } peers { permanent_uuid: "5f8ee3a3bfa8400f8751c64e4e48a6f2" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 36247 } } peers { permanent_uuid: "de84eb073d43405c9573a6db052549a0" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 42399 } }
I20260810 22:33:41.825130 18165 heartbeater.cc:503] Master 127.16.233.126:39507 was elected leader, sending a full tablet report...
I20260810 22:33:41.825446 18474 ts_tablet_manager.cc:1428] T 7462d9f327334621872d32bcbbdebd0a P c77ac95af6784e648571601102962e01: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260810 22:33:41.825834 18475 raft_consensus.cc:348] T 7462d9f327334621872d32bcbbdebd0a P de84eb073d43405c9573a6db052549a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77ac95af6784e648571601102962e01" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 37351 } } peers { permanent_uuid: "5f8ee3a3bfa8400f8751c64e4e48a6f2" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 36247 } } peers { permanent_uuid: "de84eb073d43405c9573a6db052549a0" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 42399 } }
I20260810 22:33:41.826125 18475 raft_consensus.cc:374] T 7462d9f327334621872d32bcbbdebd0a P de84eb073d43405c9573a6db052549a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:33:41.826176 18475 raft_consensus.cc:729] T 7462d9f327334621872d32bcbbdebd0a P de84eb073d43405c9573a6db052549a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de84eb073d43405c9573a6db052549a0, State: Initialized, Role: FOLLOWER
I20260810 22:33:41.826208 18473 raft_consensus.cc:348] T 7462d9f327334621872d32bcbbdebd0a P 5f8ee3a3bfa8400f8751c64e4e48a6f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77ac95af6784e648571601102962e01" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 37351 } } peers { permanent_uuid: "5f8ee3a3bfa8400f8751c64e4e48a6f2" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 36247 } } peers { permanent_uuid: "de84eb073d43405c9573a6db052549a0" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 42399 } }
I20260810 22:33:41.826542 18473 raft_consensus.cc:374] T 7462d9f327334621872d32bcbbdebd0a P 5f8ee3a3bfa8400f8751c64e4e48a6f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:33:41.826696 18475 consensus_queue.cc:260] T 7462d9f327334621872d32bcbbdebd0a P de84eb073d43405c9573a6db052549a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77ac95af6784e648571601102962e01" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 37351 } } peers { permanent_uuid: "5f8ee3a3bfa8400f8751c64e4e48a6f2" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 36247 } } peers { permanent_uuid: "de84eb073d43405c9573a6db052549a0" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 42399 } }
I20260810 22:33:41.826887 18479 raft_consensus.cc:482] T 7462d9f327334621872d32bcbbdebd0a P c77ac95af6784e648571601102962e01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:33:41.827095 18479 raft_consensus.cc:504] T 7462d9f327334621872d32bcbbdebd0a P c77ac95af6784e648571601102962e01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77ac95af6784e648571601102962e01" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 37351 } } peers { permanent_uuid: "5f8ee3a3bfa8400f8751c64e4e48a6f2" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 36247 } } peers { permanent_uuid: "de84eb073d43405c9573a6db052549a0" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 42399 } }
I20260810 22:33:41.827860 18475 ts_tablet_manager.cc:1428] T 7462d9f327334621872d32bcbbdebd0a P de84eb073d43405c9573a6db052549a0: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260810 22:33:41.828269 18309 heartbeater.cc:503] Master 127.16.233.126:39507 was elected leader, sending a full tablet report...
I20260810 22:33:41.826608 18473 raft_consensus.cc:729] T 7462d9f327334621872d32bcbbdebd0a P 5f8ee3a3bfa8400f8751c64e4e48a6f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8ee3a3bfa8400f8751c64e4e48a6f2, State: Initialized, Role: FOLLOWER
I20260810 22:33:41.829726 18473 consensus_queue.cc:260] T 7462d9f327334621872d32bcbbdebd0a P 5f8ee3a3bfa8400f8751c64e4e48a6f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77ac95af6784e648571601102962e01" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 37351 } } peers { permanent_uuid: "5f8ee3a3bfa8400f8751c64e4e48a6f2" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 36247 } } peers { permanent_uuid: "de84eb073d43405c9573a6db052549a0" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 42399 } }
I20260810 22:33:41.829913 18479 leader_election.cc:288] T 7462d9f327334621872d32bcbbdebd0a P c77ac95af6784e648571601102962e01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f8ee3a3bfa8400f8751c64e4e48a6f2 (127.16.233.67:36247), de84eb073d43405c9573a6db052549a0 (127.16.233.66:42399)
I20260810 22:33:41.831871 18456 heartbeater.cc:503] Master 127.16.233.126:39507 was elected leader, sending a full tablet report...
I20260810 22:33:41.832113 18473 ts_tablet_manager.cc:1428] T 7462d9f327334621872d32bcbbdebd0a P 5f8ee3a3bfa8400f8751c64e4e48a6f2: Time spent starting tablet: real 0.012s user 0.008s sys 0.000s
I20260810 22:33:41.834860 18470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:41.829142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:52171 (local address 127.16.233.67:36247)
0810 22:33:41.829601 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:41.829607 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:41.830287 (+ 680us) server_negotiation.cc:407] Connection header received
0810 22:33:41.830423 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:41.830426 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:41.830482 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:41.830556 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:33:41.831052 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:41.831787 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:41.833520 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:41.834130 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:41.834256 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:41.834309 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:41.834625 (+ 316us) server_negotiation.cc:299] Negotiation successful
0810 22:33:41.834740 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366}
I20260810 22:33:41.835003 18482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:41.828584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.67:36247 (local address 127.16.233.65:52171)
0810 22:33:41.830180 (+ 1596us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:41.830193 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:41.830332 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:41.830695 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:41.830698 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:41.830724 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:33:41.830918 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:41.830931 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:41.832050 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:41.832054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:41.833382 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:41.833392 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:41.834262 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:41.834268 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:41.834463 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:41.834490 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 22:33:41.834604 (+ 114us) client_negotiation.cc:240] Negotiation successful
0810 22:33:41.834678 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1525,"mutex_wait_us":20,"spinlock_wait_cycles":6912,"thread_start_us":193,"threads_started":1}
I20260810 22:33:41.836239 18407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7462d9f327334621872d32bcbbdebd0a" candidate_uuid: "c77ac95af6784e648571601102962e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8ee3a3bfa8400f8751c64e4e48a6f2" is_pre_election: true
I20260810 22:33:41.836687 18407 raft_consensus.cc:2454] T 7462d9f327334621872d32bcbbdebd0a P 5f8ee3a3bfa8400f8751c64e4e48a6f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c77ac95af6784e648571601102962e01 in term 0.
I20260810 22:33:41.837507 18483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:41.828584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.66:42399 (local address 127.16.233.65:46709)
0810 22:33:41.830565 (+ 1981us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:41.830579 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:41.830723 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:41.831210 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:41.831213 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:41.831229 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:33:41.831950 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:41.831963 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:41.833209 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:41.833212 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:41.834657 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:41.834677 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:41.837024 (+ 2347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:41.837028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:41.837130 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:41.837172 (+ 42us) client_negotiation.cc:769] Sending connection context
0810 22:33:41.837256 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 22:33:41.837338 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1870,"mutex_wait_us":23,"spinlock_wait_cycles":768,"thread_start_us":216,"threads_started":1}
I20260810 22:33:41.837592 18472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:41.829324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:46709 (local address 127.16.233.66:42399)
0810 22:33:41.829703 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:41.829708 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:41.830657 (+ 949us) server_negotiation.cc:407] Connection header received
0810 22:33:41.830964 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:41.830970 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:41.831033 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:41.831084 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:33:41.832092 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:41.832891 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:41.834815 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:41.835623 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:41.835755 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:41.835819 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:41.837331 (+ 1512us) server_negotiation.cc:299] Negotiation successful
0810 22:33:41.837432 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300}
I20260810 22:33:41.837566 18045 leader_election.cc:302] T 7462d9f327334621872d32bcbbdebd0a P c77ac95af6784e648571601102962e01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f8ee3a3bfa8400f8751c64e4e48a6f2, c77ac95af6784e648571601102962e01; no voters:
I20260810 22:33:41.837952 18480 raft_consensus.cc:2781] T 7462d9f327334621872d32bcbbdebd0a P c77ac95af6784e648571601102962e01 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:33:41.838106 18480 raft_consensus.cc:482] T 7462d9f327334621872d32bcbbdebd0a P c77ac95af6784e648571601102962e01 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:33:41.838164 18480 raft_consensus.cc:3037] T 7462d9f327334621872d32bcbbdebd0a P c77ac95af6784e648571601102962e01 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:33:41.838421 18262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7462d9f327334621872d32bcbbdebd0a" candidate_uuid: "c77ac95af6784e648571601102962e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de84eb073d43405c9573a6db052549a0" is_pre_election: true
I20260810 22:33:41.838784 18262 raft_consensus.cc:2454] T 7462d9f327334621872d32bcbbdebd0a P de84eb073d43405c9573a6db052549a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c77ac95af6784e648571601102962e01 in term 0.
I20260810 22:33:41.839936 18480 raft_consensus.cc:504] T 7462d9f327334621872d32bcbbdebd0a P c77ac95af6784e648571601102962e01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77ac95af6784e648571601102962e01" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 37351 } } peers { permanent_uuid: "5f8ee3a3bfa8400f8751c64e4e48a6f2" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 36247 } } peers { permanent_uuid: "de84eb073d43405c9573a6db052549a0" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 42399 } }
I20260810 22:33:41.840358 18480 leader_election.cc:288] T 7462d9f327334621872d32bcbbdebd0a P c77ac95af6784e648571601102962e01 [CANDIDATE]: Term 1 election: Requested vote from peers 5f8ee3a3bfa8400f8751c64e4e48a6f2 (127.16.233.67:36247), de84eb073d43405c9573a6db052549a0 (127.16.233.66:42399)
I20260810 22:33:41.840838 18262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7462d9f327334621872d32bcbbdebd0a" candidate_uuid: "c77ac95af6784e648571601102962e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de84eb073d43405c9573a6db052549a0"
I20260810 22:33:41.840905 18407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7462d9f327334621872d32bcbbdebd0a" candidate_uuid: "c77ac95af6784e648571601102962e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f8ee3a3bfa8400f8751c64e4e48a6f2"
I20260810 22:33:41.840991 18262 raft_consensus.cc:3037] T 7462d9f327334621872d32bcbbdebd0a P de84eb073d43405c9573a6db052549a0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:33:41.841045 18407 raft_consensus.cc:3037] T 7462d9f327334621872d32bcbbdebd0a P 5f8ee3a3bfa8400f8751c64e4e48a6f2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:33:41.842902 18262 raft_consensus.cc:2454] T 7462d9f327334621872d32bcbbdebd0a P de84eb073d43405c9573a6db052549a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c77ac95af6784e648571601102962e01 in term 1.
I20260810 22:33:41.842902 18407 raft_consensus.cc:2454] T 7462d9f327334621872d32bcbbdebd0a P 5f8ee3a3bfa8400f8751c64e4e48a6f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c77ac95af6784e648571601102962e01 in term 1.
I20260810 22:33:41.843312 18043 leader_election.cc:302] T 7462d9f327334621872d32bcbbdebd0a P c77ac95af6784e648571601102962e01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c77ac95af6784e648571601102962e01, de84eb073d43405c9573a6db052549a0; no voters:
I20260810 22:33:41.843550 18480 raft_consensus.cc:2781] T 7462d9f327334621872d32bcbbdebd0a P c77ac95af6784e648571601102962e01 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:33:41.843684 18480 raft_consensus.cc:686] T 7462d9f327334621872d32bcbbdebd0a P c77ac95af6784e648571601102962e01 [term 1 LEADER]: Becoming Leader. State: Replica: c77ac95af6784e648571601102962e01, State: Running, Role: LEADER
I20260810 22:33:41.844115 18480 consensus_queue.cc:237] T 7462d9f327334621872d32bcbbdebd0a P c77ac95af6784e648571601102962e01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77ac95af6784e648571601102962e01" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 37351 } } peers { permanent_uuid: "5f8ee3a3bfa8400f8751c64e4e48a6f2" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 36247 } } peers { permanent_uuid: "de84eb073d43405c9573a6db052549a0" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 42399 } }
I20260810 22:33:41.848006 17952 catalog_manager.cc:5495] T 7462d9f327334621872d32bcbbdebd0a P c77ac95af6784e648571601102962e01 reported cstate change: term changed from 0 to 1, leader changed from <none> to c77ac95af6784e648571601102962e01 (127.16.233.65). New cstate: current_term: 1 leader_uuid: "c77ac95af6784e648571601102962e01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77ac95af6784e648571601102962e01" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 37351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f8ee3a3bfa8400f8751c64e4e48a6f2" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 36247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de84eb073d43405c9573a6db052549a0" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 42399 } health_report { overall_health: UNKNOWN } } }
Aug 10 22:33:41 dist-test-slave-77v9 krb5kdc[17899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.65@KRBTEST.COM
I20260810 22:33:41.869674 18469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:41.857826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56308 (local address 127.16.233.65:37351)
0810 22:33:41.857997 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:41.858003 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:41.858369 (+ 366us) server_negotiation.cc:407] Connection header received
0810 22:33:41.858551 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:41.858555 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:41.858611 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:41.858704 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:41.859501 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:41.860025 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:41.861132 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:41.861414 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:41.861528 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:41.865115 (+ 3587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:41.865131 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:41.865146 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:41.865178 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:41.867376 (+ 2198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:41.868066 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:41.868073 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:41.868074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:41.868168 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:41.868491 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:41.868494 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:41.868495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:41.868936 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:41.869126 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:41.869336 (+ 210us) server_negotiation.cc:299] Negotiation successful
0810 22:33:41.869448 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
Aug 10 22:33:41 dist-test-slave-77v9 krb5kdc[17899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401221, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 22:33:41 dist-test-slave-77v9 krb5kdc[17899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401221, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.233.65@KRBTEST.COM
I20260810 22:33:41.904130 18469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:41.894360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56316 (local address 127.16.233.65:37351)
0810 22:33:41.894559 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:41.894564 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:41.894709 (+ 145us) server_negotiation.cc:407] Connection header received
0810 22:33:41.894953 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:41.894962 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:41.895009 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:41.895107 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:41.895990 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:41.896594 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:41.897699 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:41.897978 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:41.898101 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:41.900768 (+ 2667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:41.900771 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:41.900773 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:41.900800 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:41.902239 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:41.902731 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:41.902736 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:41.902738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:41.902824 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:41.903116 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:41.903121 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:41.903122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:41.903350 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:41.903505 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:41.903817 (+ 312us) server_negotiation.cc:299] Negotiation successful
0810 22:33:41.903948 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
Aug 10 22:33:41 dist-test-slave-77v9 krb5kdc[17899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401221, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 22:33:41 dist-test-slave-77v9 krb5kdc[17899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401221, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.65@KRBTEST.COM
I20260810 22:33:41.934621 18469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:41.924798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56318 (local address 127.16.233.65:37351)
0810 22:33:41.924998 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:41.925005 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:41.925242 (+ 237us) server_negotiation.cc:407] Connection header received
0810 22:33:41.925447 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:41.925451 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:41.925502 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:41.925596 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:41.926383 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:41.926982 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:41.927863 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:41.928120 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:41.928248 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:41.931029 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:41.931032 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:41.931034 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:41.931061 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:41.932706 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:41.933325 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:41.933329 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:41.933331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:41.933419 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:41.933704 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:41.933709 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:41.933711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:41.933899 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:41.934042 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:41.934317 (+ 275us) server_negotiation.cc:299] Negotiation successful
0810 22:33:41.934444 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260810 22:33:41.936244 18071 tablet_service.cc:2908] Rejecting scan request for tablet 7462d9f327334621872d32bcbbdebd0a: Uninitialized: clean time has not yet been initialized
I20260810 22:33:41.938952 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 18024
I20260810 22:33:41.950088 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 18170
I20260810 22:33:41.958513 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 18316
I20260810 22:33:41.966369 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 17915
2026-08-10T22:33:41Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2566 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:33:42 dist-test-slave-77v9 krb5kdc[18499](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:33:42 dist-test-slave-77v9 krb5kdc[18499](info): set up 2 sockets
Aug 10 22:33:42 dist-test-slave-77v9 krb5kdc[18499](info): commencing operation
krb5kdc: starting...
W20260810 22:33:43.107828 17317 mini_kdc.cc:121] Time spent starting KDC: real 1.114s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Aug 10 22:33:43 dist-test-slave-77v9 krb5kdc[18499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:33:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:33:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.233.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab.
Entry for principal kudu/127.16.233.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab.
I20260810 22:33:43.257362 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:43649
--webserver_interface=127.16.233.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab
--principal=kudu/127.16.233.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:45101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.126:43649
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:33:43.375999 18516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:33:43.376363 18516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:33:43.376443 18516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:33:43.380663 18516 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:33:43.380789 18516 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 22:33:43.380826 18516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:33:43.380884 18516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:33:43.380920 18516 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:33:43.380946 18516 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:33:43.386822 18516 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:45101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.126:43649
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab
--principal=kudu/127.16.233.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:43649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:33:43.388053 18516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:33:43.389205 18516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 22:33:43.398501 18516 server_base.cc:1034] running on GCE node
W20260810 22:33:43.398509 18523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:33:43.398500 18524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:33:43.398496 18526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:33:43.399380 18516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:33:43.400575 18516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:33:43.401873 18516 hybrid_clock.cc:648] HybridClock initialized: now 1786401223401868 us; error 54 us; skew 500 ppm
Aug 10 22:33:43 dist-test-slave-77v9 krb5kdc[18499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401223, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:33:43.405241 18516 init.cc:377] Logged in from keytab as kudu/127.16.233.126@KRBTEST.COM (short username kudu)
I20260810 22:33:43.406425 18516 webserver.cc:460] Webserver started at http://127.16.233.126:36879/ using document root <none> and password file <none>
I20260810 22:33:43.407112 18516 fs_manager.cc:359] Metadata directory not provided
I20260810 22:33:43.407225 18516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:33:43.407490 18516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:33:43.410115 18516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/master-0/data/instance:
uuid: "04121952a88449349c980197e530ae20"
format_stamp: "Formatted at 2026-08-10 22:33:43 on dist-test-slave-77v9"
server_key: "79d01b4a9a6e18571fdb48b0c2da7019"
server_key_iv: "17d428cb946069c74673ce1596f6846d"
server_key_version: "encryptionkey@0"
I20260810 22:33:43.410840 18516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/master-0/wal/instance:
uuid: "04121952a88449349c980197e530ae20"
format_stamp: "Formatted at 2026-08-10 22:33:43 on dist-test-slave-77v9"
server_key: "79d01b4a9a6e18571fdb48b0c2da7019"
server_key_iv: "17d428cb946069c74673ce1596f6846d"
server_key_version: "encryptionkey@0"
I20260810 22:33:43.415612 18516 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 22:33:43.418671 18534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:43.420187 18516 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 22:33:43.420352 18516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/master-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/master-0/wal
uuid: "04121952a88449349c980197e530ae20"
format_stamp: "Formatted at 2026-08-10 22:33:43 on dist-test-slave-77v9"
server_key: "79d01b4a9a6e18571fdb48b0c2da7019"
server_key_iv: "17d428cb946069c74673ce1596f6846d"
server_key_version: "encryptionkey@0"
I20260810 22:33:43.420490 18516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:33:43.460379 18516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:33:43.464031 18516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:33:43.464275 18516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:33:43.473793 18516 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.126:43649
I20260810 22:33:43.473815 18602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.126:43649 every 8 connection(s)
I20260810 22:33:43.474845 18516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
I20260810 22:33:43.476274 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 18516
I20260810 22:33:43.476421 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/master-0/wal/instance
I20260810 22:33:43.476759 17317 external_mini_cluster.cc:1428] Setting key 53fa3160b044327d35f1629ae8f05a33
I20260810 22:33:43.478721 18603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:33:43.485102 18603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04121952a88449349c980197e530ae20: Bootstrap starting.
Aug 10 22:33:43 dist-test-slave-77v9 krb5kdc[18499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:33:43.487977 18603 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04121952a88449349c980197e530ae20: Neither blocks nor log segments found. Creating new log.
I20260810 22:33:43.489480 18603 log.cc:824] T 00000000000000000000000000000000 P 04121952a88449349c980197e530ae20: Log is configured to *not* fsync() on all Append() calls
I20260810 22:33:43.491676 18603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04121952a88449349c980197e530ae20: No bootstrap required, opened a new log
I20260810 22:33:43.492949 18606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:43.478534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40368 (local address 127.16.233.126:43649)
0810 22:33:43.479115 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:43.479133 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:43.479173 (+ 40us) server_negotiation.cc:407] Connection header received
0810 22:33:43.479937 (+ 764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:43.479957 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:43.480514 (+ 557us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:43.480984 (+ 470us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:43.481840 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:43.482834 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:43.483743 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:43.484247 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:43.484440 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:43.487710 (+ 3270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:43.487720 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:43.487738 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:43.487783 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:43.489748 (+ 1965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:43.490302 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:43.490308 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:43.490313 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:43.490444 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:43.490759 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:43.490763 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:43.490765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:43.491211 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:43.491435 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:43.491739 (+ 304us) server_negotiation.cc:299] Negotiation successful
0810 22:33:43.492086 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":204,"threads_started":1}
I20260810 22:33:43.495210 18603 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04121952a88449349c980197e530ae20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04121952a88449349c980197e530ae20" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 43649 } }
I20260810 22:33:43.495532 18603 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04121952a88449349c980197e530ae20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:33:43.495617 18603 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04121952a88449349c980197e530ae20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04121952a88449349c980197e530ae20, State: Initialized, Role: FOLLOWER
I20260810 22:33:43.496061 18603 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04121952a88449349c980197e530ae20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04121952a88449349c980197e530ae20" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 43649 } }
I20260810 22:33:43.496234 18603 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04121952a88449349c980197e530ae20 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:33:43.496335 18603 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04121952a88449349c980197e530ae20 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:33:43.496459 18603 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04121952a88449349c980197e530ae20 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:33:43.497612 18603 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04121952a88449349c980197e530ae20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04121952a88449349c980197e530ae20" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 43649 } }
I20260810 22:33:43.497996 18603 leader_election.cc:302] T 00000000000000000000000000000000 P 04121952a88449349c980197e530ae20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04121952a88449349c980197e530ae20; no voters:
I20260810 22:33:43.498373 18603 leader_election.cc:288] T 00000000000000000000000000000000 P 04121952a88449349c980197e530ae20 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:33:43.498564 18608 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04121952a88449349c980197e530ae20 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:33:43.498811 18608 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04121952a88449349c980197e530ae20 [term 1 LEADER]: Becoming Leader. State: Replica: 04121952a88449349c980197e530ae20, State: Running, Role: LEADER
I20260810 22:33:43.499140 18608 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04121952a88449349c980197e530ae20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04121952a88449349c980197e530ae20" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 43649 } }
I20260810 22:33:43.499752 18603 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04121952a88449349c980197e530ae20 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:33:43.500969 18611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04121952a88449349c980197e530ae20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04121952a88449349c980197e530ae20. Latest consensus state: current_term: 1 leader_uuid: "04121952a88449349c980197e530ae20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04121952a88449349c980197e530ae20" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 43649 } } }
I20260810 22:33:43.501019 18610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04121952a88449349c980197e530ae20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04121952a88449349c980197e530ae20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04121952a88449349c980197e530ae20" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 43649 } } }
I20260810 22:33:43.501117 18611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04121952a88449349c980197e530ae20 [sys.catalog]: This master's current role is: LEADER
I20260810 22:33:43.501129 18610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04121952a88449349c980197e530ae20 [sys.catalog]: This master's current role is: LEADER
I20260810 22:33:43.501555 18620 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:33:43.504961 18620 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:33:43.512071 18620 catalog_manager.cc:1347] Generated new cluster ID: 13741d84f04344e69f36a3d9c8416141
I20260810 22:33:43.512181 18620 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:33:43.529965 18620 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:33:43.531111 18620 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:33:43.541512 18620 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04121952a88449349c980197e530ae20: Generated new TSK 0
I20260810 22:33:43.542385 18620 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.233.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab.
Entry for principal kudu/127.16.233.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab.
I20260810 22:33:43.589247 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.65:0
--local_ip_for_outbound_sockets=127.16.233.65
--webserver_interface=127.16.233.65
--webserver_port=0
--tserver_master_addrs=127.16.233.126:43649
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab
--principal=kudu/127.16.233.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:45101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:33:43.730542 18632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:33:43.730875 18632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:33:43.730929 18632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:33:43.735134 18632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:33:43.735231 18632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:33:43.735277 18632 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 22:33:43.735316 18632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.65
I20260810 22:33:43.741899 18632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:45101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab
--principal=kudu/127.16.233.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.233.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.233.126:43649
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.65
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:33:43.743139 18632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:33:43.744302 18632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:33:43.755529 18637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:33:43.755540 18638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:33:43.755530 18640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:33:43.755942 18632 server_base.cc:1034] running on GCE node
I20260810 22:33:43.756594 18632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:33:43.757396 18632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:33:43.758631 18632 hybrid_clock.cc:648] HybridClock initialized: now 1786401223758609 us; error 32 us; skew 500 ppm
Aug 10 22:33:43 dist-test-slave-77v9 krb5kdc[18499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401223, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:33:43.762331 18632 init.cc:377] Logged in from keytab as kudu/127.16.233.65@KRBTEST.COM (short username kudu)
I20260810 22:33:43.763401 18632 webserver.cc:460] Webserver started at http://127.16.233.65:43271/ using document root <none> and password file <none>
I20260810 22:33:43.763989 18632 fs_manager.cc:359] Metadata directory not provided
I20260810 22:33:43.764060 18632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:33:43.764245 18632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:33:43.766677 18632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/ts-0/data/instance:
uuid: "c1914598e5c74226b1200a6f2c694c8f"
format_stamp: "Formatted at 2026-08-10 22:33:43 on dist-test-slave-77v9"
server_key: "d937bbab7b906364fa9d65e2ef280d08"
server_key_iv: "9a7ef5ff728d2fd1f956130e64b85cdb"
server_key_version: "encryptionkey@0"
I20260810 22:33:43.767288 18632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/ts-0/wal/instance:
uuid: "c1914598e5c74226b1200a6f2c694c8f"
format_stamp: "Formatted at 2026-08-10 22:33:43 on dist-test-slave-77v9"
server_key: "d937bbab7b906364fa9d65e2ef280d08"
server_key_iv: "9a7ef5ff728d2fd1f956130e64b85cdb"
server_key_version: "encryptionkey@0"
I20260810 22:33:43.771731 18632 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260810 22:33:43.774596 18647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:43.776063 18632 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 22:33:43.776223 18632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/ts-0/wal
uuid: "c1914598e5c74226b1200a6f2c694c8f"
format_stamp: "Formatted at 2026-08-10 22:33:43 on dist-test-slave-77v9"
server_key: "d937bbab7b906364fa9d65e2ef280d08"
server_key_iv: "9a7ef5ff728d2fd1f956130e64b85cdb"
server_key_version: "encryptionkey@0"
I20260810 22:33:43.776372 18632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:33:43.795809 18632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:33:43.799769 18632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:33:43.799993 18632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:33:43.801918 18632 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:33:43.801981 18632 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:43.802026 18632 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:33:43.802078 18632 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 22:33:43 dist-test-slave-77v9 krb5kdc[18499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401223, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.65@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:33:43.813656 18632 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.65:35253
I20260810 22:33:43.813750 18773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.65:35253 every 8 connection(s)
I20260810 22:33:43.814677 18632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
I20260810 22:33:43.818390 18606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:43.802965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:47471 (local address 127.16.233.126:43649)
0810 22:33:43.803154 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:43.803160 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:43.804161 (+ 1001us) server_negotiation.cc:407] Connection header received
0810 22:33:43.805533 (+ 1372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:43.805536 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:43.805605 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:43.805709 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:43.807565 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:43.808194 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:43.810222 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:43.810530 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:43.810632 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:43.814050 (+ 3418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:43.814053 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:43.814056 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:43.814087 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:43.816024 (+ 1937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:43.816656 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:43.816659 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:43.816660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:43.816749 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:43.817226 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:43.817229 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:43.817231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:43.817477 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:43.817572 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:43.818091 (+ 519us) server_negotiation.cc:299] Negotiation successful
0810 22:33:43.818220 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260810 22:33:43.819229 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 18632
I20260810 22:33:43.819314 18656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:43.803450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:43649 (local address 127.16.233.65:47471)
0810 22:33:43.803979 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:43.804039 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:43.805282 (+ 1243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:43.805948 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:43.805957 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:43.806532 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:33:43.807349 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:43.807375 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:43.808352 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:43.808356 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:43.810050 (+ 1694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:43.810066 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:43.810735 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:43.810742 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:43.810972 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:43.811870 (+ 898us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:33:43.811898 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:33:43.813837 (+ 1939us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:33:43.816188 (+ 2351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:43.816197 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:43.816212 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:43.816543 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:43.816963 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:43.816966 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:43.816967 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:43.817121 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:43.817583 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:33:43.817587 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:33:43.817848 (+ 261us) client_negotiation.cc:769] Sending connection context
0810 22:33:43.818003 (+ 155us) client_negotiation.cc:240] Negotiation successful
0810 22:33:43.818293 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":142,"threads_started":1}
I20260810 22:33:43.819415 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/ts-0/wal/instance
I20260810 22:33:43.819769 17317 external_mini_cluster.cc:1428] Setting key f31d918151ba494ed0b74fc8c5022722
I20260810 22:33:43.821089 18774 heartbeater.cc:347] Connected to a master server at 127.16.233.126:43649
I20260810 22:33:43.821285 18774 heartbeater.cc:464] Registering TS with master...
I20260810 22:33:43.821849 18774 heartbeater.cc:511] Master 127.16.233.126:43649 requested a full tablet report, sending...
I20260810 22:33:43.823529 18547 ts_manager.cc:194] Registered new tserver with Master: c1914598e5c74226b1200a6f2c694c8f (127.16.233.65:35253)
I20260810 22:33:43.825043 18547 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.65@KRBTEST.COM'} at 127.16.233.65:47471
WARNING: no policy specified for kudu/127.16.233.66@KRBTEST.COM; defaulting to no policy
I20260810 22:33:43.834589 18781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:43.823654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:43649 (local address 127.16.233.65:33217)
0810 22:33:43.824023 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:43.824044 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:43.824157 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:43.825420 (+ 1263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:43.825424 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:43.825819 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:33:43.825962 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:43.825970 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:43.827260 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:43.827264 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:43.827899 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:43.827909 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:43.828866 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:43.828875 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:43.828975 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:43.829636 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:33:43.829650 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:33:43.830128 (+ 478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:33:43.832496 (+ 2368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:43.832499 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:43.832500 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:43.832914 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:43.833308 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:43.833311 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:43.833312 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:43.833414 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:43.833951 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:33:43.833953 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:33:43.834131 (+ 178us) client_negotiation.cc:769] Sending connection context
0810 22:33:43.834273 (+ 142us) client_negotiation.cc:240] Negotiation successful
0810 22:33:43.834430 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":143,"threads_started":1}
I20260810 22:33:43.834625 18606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:43.824945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:33217 (local address 127.16.233.126:43649)
0810 22:33:43.825105 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:43.825113 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:43.825131 (+ 18us) server_negotiation.cc:407] Connection header received
0810 22:33:43.825179 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:43.825185 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:43.825250 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:43.825358 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:43.826108 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:43.827079 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:43.828035 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:43.828560 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:43.828711 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:43.830314 (+ 1603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:43.830318 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:43.830322 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:43.830361 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:43.832354 (+ 1993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:43.833041 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:43.833048 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:43.833051 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:43.833159 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:43.833517 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:43.833520 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:43.833521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:43.833719 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:43.833807 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:43.834388 (+ 581us) server_negotiation.cc:299] Negotiation successful
0810 22:33:43.834507 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab.
Entry for principal kudu/127.16.233.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab.
I20260810 22:33:43.858054 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.66:0
--local_ip_for_outbound_sockets=127.16.233.66
--webserver_interface=127.16.233.66
--webserver_port=0
--tserver_master_addrs=127.16.233.126:43649
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab
--principal=kudu/127.16.233.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:45101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:33:43.984537 18784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:33:43.984876 18784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:33:43.984929 18784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:33:43.989184 18784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:33:43.989269 18784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:33:43.989310 18784 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 22:33:43.989343 18784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.66
I20260810 22:33:43.994935 18784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:45101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab
--principal=kudu/127.16.233.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.233.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.233.126:43649
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.66
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:33:43.996222 18784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:33:43.997325 18784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:33:44.008081 18791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:33:44.008060 18790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:33:44.008148 18784 server_base.cc:1034] running on GCE node
W20260810 22:33:44.008055 18793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:33:44.008922 18784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:33:44.009665 18784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:33:44.010957 18784 hybrid_clock.cc:648] HybridClock initialized: now 1786401224010910 us; error 58 us; skew 500 ppm
Aug 10 22:33:44 dist-test-slave-77v9 krb5kdc[18499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401224, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:33:44.014433 18784 init.cc:377] Logged in from keytab as kudu/127.16.233.66@KRBTEST.COM (short username kudu)
I20260810 22:33:44.015435 18784 webserver.cc:460] Webserver started at http://127.16.233.66:39373/ using document root <none> and password file <none>
I20260810 22:33:44.016042 18784 fs_manager.cc:359] Metadata directory not provided
I20260810 22:33:44.016106 18784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:33:44.016299 18784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:33:44.018695 18784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/ts-1/data/instance:
uuid: "d4f02cff8e1a4d8f98e81b5b2e80becc"
format_stamp: "Formatted at 2026-08-10 22:33:44 on dist-test-slave-77v9"
server_key: "743e02d3a30c529b1c4ec5b7dddea62b"
server_key_iv: "c81c72a717661e4f5c3519fa38536efd"
server_key_version: "encryptionkey@0"
I20260810 22:33:44.019285 18784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/ts-1/wal/instance:
uuid: "d4f02cff8e1a4d8f98e81b5b2e80becc"
format_stamp: "Formatted at 2026-08-10 22:33:44 on dist-test-slave-77v9"
server_key: "743e02d3a30c529b1c4ec5b7dddea62b"
server_key_iv: "c81c72a717661e4f5c3519fa38536efd"
server_key_version: "encryptionkey@0"
I20260810 22:33:44.023710 18784 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260810 22:33:44.026823 18800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:44.028381 18784 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 22:33:44.028508 18784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/ts-1/wal
uuid: "d4f02cff8e1a4d8f98e81b5b2e80becc"
format_stamp: "Formatted at 2026-08-10 22:33:44 on dist-test-slave-77v9"
server_key: "743e02d3a30c529b1c4ec5b7dddea62b"
server_key_iv: "c81c72a717661e4f5c3519fa38536efd"
server_key_version: "encryptionkey@0"
I20260810 22:33:44.028620 18784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:33:44.047024 18784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:33:44.050772 18784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:33:44.050977 18784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:33:44.052546 18784 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:33:44.052609 18784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:44.052656 18784 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:33:44.052753 18784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 22:33:44 dist-test-slave-77v9 krb5kdc[18499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401224, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.66@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:33:44.065912 18784 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.66:39683
I20260810 22:33:44.067057 18784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
I20260810 22:33:44.068102 18926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.66:39683 every 8 connection(s)
I20260810 22:33:44.068513 18606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:44.053851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:43889 (local address 127.16.233.126:43649)
0810 22:33:44.054030 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:44.054035 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:44.054952 (+ 917us) server_negotiation.cc:407] Connection header received
0810 22:33:44.056219 (+ 1267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:44.056222 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:44.056286 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:44.056413 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:44.058267 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:44.058918 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:44.060200 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:44.060485 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:44.060596 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:44.063839 (+ 3243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:44.063843 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:44.063845 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:44.063878 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:44.065898 (+ 2020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:44.066560 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:44.066564 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:44.066566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:44.066682 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:44.067134 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:44.067137 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:44.067139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:44.067402 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:44.067528 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:44.068172 (+ 644us) server_negotiation.cc:299] Negotiation successful
0810 22:33:44.068319 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 22:33:44.069399 18809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:44.054315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:43649 (local address 127.16.233.66:43889)
0810 22:33:44.054790 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:44.054843 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:44.055979 (+ 1136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:44.056652 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:44.056664 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:44.057287 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:33:44.058057 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:44.058081 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:44.059069 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:44.059072 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:44.060048 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:44.060064 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:44.060651 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:44.060656 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:44.060937 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:44.061642 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:33:44.061660 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:33:44.063582 (+ 1922us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:33:44.066088 (+ 2506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:44.066095 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:44.066111 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:44.066413 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:44.066794 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:44.066797 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:44.066799 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:44.067020 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:44.067529 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:33:44.067537 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:33:44.067838 (+ 301us) client_negotiation.cc:769] Sending connection context
0810 22:33:44.068121 (+ 283us) client_negotiation.cc:240] Negotiation successful
0810 22:33:44.068462 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":143,"threads_started":1}
I20260810 22:33:44.070871 18927 heartbeater.cc:347] Connected to a master server at 127.16.233.126:43649
I20260810 22:33:44.071095 18927 heartbeater.cc:464] Registering TS with master...
I20260810 22:33:44.071661 18927 heartbeater.cc:511] Master 127.16.233.126:43649 requested a full tablet report, sending...
I20260810 22:33:44.072919 18547 ts_manager.cc:194] Registered new tserver with Master: d4f02cff8e1a4d8f98e81b5b2e80becc (127.16.233.66:39683)
I20260810 22:33:44.073649 18547 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.66@KRBTEST.COM'} at 127.16.233.66:43889
I20260810 22:33:44.077287 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 18784
I20260810 22:33:44.077391 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/ts-1/wal/instance
I20260810 22:33:44.077708 17317 external_mini_cluster.cc:1428] Setting key 5e1428f9892678b13664ef9df7f48c01
I20260810 22:33:44.081693 18606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:44.073304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:44401 (local address 127.16.233.126:43649)
0810 22:33:44.073449 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:44.073455 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:44.073658 (+ 203us) server_negotiation.cc:407] Connection header received
0810 22:33:44.073798 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:44.073801 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:44.073850 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:44.073938 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:44.074439 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:44.075040 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:44.076131 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:44.076355 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:44.076449 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:44.077936 (+ 1487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:44.077939 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:44.077941 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:44.077965 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:44.079712 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:44.080257 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:44.080261 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:44.080263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:44.080347 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:44.080662 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:44.080666 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:44.080668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:44.080984 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:44.081097 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:44.081416 (+ 319us) server_negotiation.cc:299] Negotiation successful
0810 22:33:44.081533 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 22:33:44.081707 18934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:44.073228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:43649 (local address 127.16.233.66:44401)
0810 22:33:44.073573 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:44.073585 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:44.073702 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:44.073955 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:44.073957 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:44.074173 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:33:44.074312 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:44.074320 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:44.075187 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:44.075191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:44.075974 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:44.075984 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:44.076493 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:44.076497 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:44.076587 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:44.077243 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:33:44.077254 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:33:44.077726 (+ 472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:33:44.079848 (+ 2122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:44.079851 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:44.079853 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:44.080086 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:44.080450 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:44.080453 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:44.080455 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:44.080550 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:44.081087 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:33:44.081090 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:33:44.081195 (+ 105us) client_negotiation.cc:769] Sending connection context
0810 22:33:44.081352 (+ 157us) client_negotiation.cc:240] Negotiation successful
0810 22:33:44.081522 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":114,"threads_started":1}
WARNING: no policy specified for kudu/127.16.233.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab.
Entry for principal kudu/127.16.233.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab.
I20260810 22:33:44.113550 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.67:0
--local_ip_for_outbound_sockets=127.16.233.67
--webserver_interface=127.16.233.67
--webserver_port=0
--tserver_master_addrs=127.16.233.126:43649
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab
--principal=kudu/127.16.233.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:45101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:33:44.233057 18937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:33:44.233415 18937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:33:44.233525 18937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:33:44.237931 18937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:33:44.238022 18937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:33:44.238065 18937 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 22:33:44.238101 18937 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.67
I20260810 22:33:44.243860 18937 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:45101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab
--principal=kudu/127.16.233.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.233.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.233.126:43649
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18937
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.67
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:33:44.245185 18937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:33:44.246297 18937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 22:33:44.256803 18937 server_base.cc:1034] running on GCE node
W20260810 22:33:44.256716 18942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:33:44.256716 18943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:33:44.256750 18945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:33:44.257570 18937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:33:44.258292 18937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:33:44.259470 18937 hybrid_clock.cc:648] HybridClock initialized: now 1786401224259456 us; error 28 us; skew 500 ppm
Aug 10 22:33:44 dist-test-slave-77v9 krb5kdc[18499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401224, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:33:44.262638 18937 init.cc:377] Logged in from keytab as kudu/127.16.233.67@KRBTEST.COM (short username kudu)
I20260810 22:33:44.263630 18937 webserver.cc:460] Webserver started at http://127.16.233.67:46733/ using document root <none> and password file <none>
I20260810 22:33:44.264168 18937 fs_manager.cc:359] Metadata directory not provided
I20260810 22:33:44.264230 18937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:33:44.264413 18937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:33:44.266631 18937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/ts-2/data/instance:
uuid: "83c6dfaf96b34764967f6f5497309bf1"
format_stamp: "Formatted at 2026-08-10 22:33:44 on dist-test-slave-77v9"
server_key: "9a0c4381c1b41dffa420fea58faffe03"
server_key_iv: "463eb0264a6fa40b3104c584136fe7ae"
server_key_version: "encryptionkey@0"
I20260810 22:33:44.267179 18937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/ts-2/wal/instance:
uuid: "83c6dfaf96b34764967f6f5497309bf1"
format_stamp: "Formatted at 2026-08-10 22:33:44 on dist-test-slave-77v9"
server_key: "9a0c4381c1b41dffa420fea58faffe03"
server_key_iv: "463eb0264a6fa40b3104c584136fe7ae"
server_key_version: "encryptionkey@0"
I20260810 22:33:44.271364 18937 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260810 22:33:44.274091 18952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:44.275435 18937 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 22:33:44.275583 18937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/ts-2/wal
uuid: "83c6dfaf96b34764967f6f5497309bf1"
format_stamp: "Formatted at 2026-08-10 22:33:44 on dist-test-slave-77v9"
server_key: "9a0c4381c1b41dffa420fea58faffe03"
server_key_iv: "463eb0264a6fa40b3104c584136fe7ae"
server_key_version: "encryptionkey@0"
I20260810 22:33:44.275710 18937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:33:44.293519 18937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:33:44.297555 18937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:33:44.297811 18937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:33:44.299489 18937 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:33:44.299571 18937 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:44.299635 18937 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:33:44.299690 18937 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Aug 10 22:33:44 dist-test-slave-77v9 krb5kdc[18499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401224, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.67@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:33:44.311919 18937 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.67:43165
I20260810 22:33:44.313011 19082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.67:43165 every 8 connection(s)
I20260810 22:33:44.313074 18937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
I20260810 22:33:44.314756 18606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:44.300703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:44733 (local address 127.16.233.126:43649)
0810 22:33:44.300893 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:44.300899 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:44.301942 (+ 1043us) server_negotiation.cc:407] Connection header received
0810 22:33:44.303285 (+ 1343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:44.303290 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:44.303366 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:44.303480 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:44.304950 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:44.305503 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:44.306726 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:44.306986 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:44.307105 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:44.310369 (+ 3264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:44.310373 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:44.310376 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:44.310414 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:44.312330 (+ 1916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:44.312980 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:44.312983 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:44.312985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:44.313072 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:44.313480 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:44.313483 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:44.313486 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:44.313764 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:44.313871 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:44.314501 (+ 630us) server_negotiation.cc:299] Negotiation successful
0810 22:33:44.314625 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 22:33:44.315588 18962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:44.301198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:43649 (local address 127.16.233.67:44733)
0810 22:33:44.301773 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:44.301829 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:44.303064 (+ 1235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:44.303683 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:44.303690 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:44.304067 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:33:44.304722 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:44.304743 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:44.305653 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:44.305656 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:44.306574 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:44.306584 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:44.307186 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:44.307190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:44.307403 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:44.308160 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:33:44.308191 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:33:44.310145 (+ 1954us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:33:44.312481 (+ 2336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:44.312487 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:44.312502 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:44.312784 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:44.313206 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:44.313209 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:44.313211 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:44.313366 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:44.313927 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:33:44.313935 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:33:44.314211 (+ 276us) client_negotiation.cc:769] Sending connection context
0810 22:33:44.314383 (+ 172us) client_negotiation.cc:240] Negotiation successful
0810 22:33:44.314654 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":139,"threads_started":1}
I20260810 22:33:44.317307 19083 heartbeater.cc:347] Connected to a master server at 127.16.233.126:43649
I20260810 22:33:44.317477 19083 heartbeater.cc:464] Registering TS with master...
I20260810 22:33:44.318284 19083 heartbeater.cc:511] Master 127.16.233.126:43649 requested a full tablet report, sending...
I20260810 22:33:44.319813 18548 ts_manager.cc:194] Registered new tserver with Master: 83c6dfaf96b34764967f6f5497309bf1 (127.16.233.67:43165)
I20260810 22:33:44.320762 18548 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.67@KRBTEST.COM'} at 127.16.233.67:44733
I20260810 22:33:44.322546 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 18937
I20260810 22:33:44.322695 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/ts-2/wal/instance
I20260810 22:33:44.323009 17317 external_mini_cluster.cc:1428] Setting key b02669abeb9e37d58e0ad48fa585d429
I20260810 22:33:44.325585 17317 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 22:33:44.328729 18606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:44.320309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:50635 (local address 127.16.233.126:43649)
0810 22:33:44.320462 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:44.320470 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:44.320653 (+ 183us) server_negotiation.cc:407] Connection header received
0810 22:33:44.320771 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:44.320774 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:44.320869 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:44.320970 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:44.321543 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:44.322061 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:44.323057 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:44.323286 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:44.323410 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:44.324921 (+ 1511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:44.324927 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:44.324929 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:44.324956 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:44.326733 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:44.327300 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:44.327303 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:44.327305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:44.327388 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:44.327699 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:44.327701 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:44.327702 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:44.327959 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:44.328099 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:44.328413 (+ 314us) server_negotiation.cc:299] Negotiation successful
0810 22:33:44.328540 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 22:33:44.328749 19089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:44.320162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:43649 (local address 127.16.233.67:50635)
0810 22:33:44.320552 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:44.320567 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:44.320677 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:44.320987 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:44.320990 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:44.321236 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:33:44.321395 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:44.321426 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:44.322218 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:44.322223 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:44.322928 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:44.322937 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:44.323409 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:44.323413 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:44.323505 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:44.324157 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:33:44.324171 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:33:44.324669 (+ 498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:33:44.326860 (+ 2191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:44.326867 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:44.326872 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:44.327185 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:44.327481 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:44.327487 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:44.327492 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:44.327601 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:44.328087 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:33:44.328091 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:33:44.328209 (+ 118us) client_negotiation.cc:769] Sending connection context
0810 22:33:44.328331 (+ 122us) client_negotiation.cc:240] Negotiation successful
0810 22:33:44.328533 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":144,"threads_started":1}
Aug 10 22:33:44 dist-test-slave-77v9 krb5kdc[18499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401224, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 22:33:44 dist-test-slave-77v9 krb5kdc[18499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401224, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:33:44.355883 18606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:44.346665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40372 (local address 127.16.233.126:43649)
0810 22:33:44.346802 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:44.346806 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:44.346818 (+ 12us) server_negotiation.cc:407] Connection header received
0810 22:33:44.346870 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:44.346872 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:44.346912 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:44.346981 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:44.347784 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:44.348366 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:44.349369 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:44.349590 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:44.349678 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:44.352378 (+ 2700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:44.352382 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:44.352384 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:44.352418 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:44.353989 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:44.354471 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:44.354480 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:44.354481 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:44.354563 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:44.354864 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:44.354867 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:44.354869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:44.355109 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:44.355270 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:44.355563 (+ 293us) server_negotiation.cc:299] Negotiation successful
0810 22:33:44.355690 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 22:33:44.359659 18548 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40372:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 22:33:44.363375 18548 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:33:44.379345 19103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:44.373095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40080 (local address 127.16.233.66:39683)
0810 22:33:44.373927 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:44.373941 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:44.373980 (+ 39us) server_negotiation.cc:407] Connection header received
0810 22:33:44.374065 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:44.374070 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:44.374291 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:44.374499 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:33:44.374890 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:44.376105 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:44.377948 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:44.378539 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:44.378683 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:44.378879 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:44.379047 (+ 168us) server_negotiation.cc:299] Negotiation successful
0810 22:33:44.379141 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":147,"threads_started":1}
I20260810 22:33:44.379323 19102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:44.372930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.66:39683 (local address 127.0.0.1:40080)
0810 22:33:44.373344 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:44.373404 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:44.373542 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:44.374528 (+ 986us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:44.374535 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:44.374562 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:33:44.374730 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:44.374737 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:44.376244 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:44.376248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:44.377788 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:44.377799 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:44.378700 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:44.378704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:44.378832 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:44.378879 (+ 47us) client_negotiation.cc:769] Sending connection context
0810 22:33:44.378998 (+ 119us) client_negotiation.cc:240] Negotiation successful
0810 22:33:44.379140 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":51,"thread_start_us":95,"threads_started":1}
I20260810 22:33:44.379614 19104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:44.373721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.67:43165 (local address 127.0.0.1:36986)
0810 22:33:44.374675 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:44.374693 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:44.374800 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:44.375283 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:44.375288 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:44.375308 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:33:44.375395 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:44.375401 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:44.376735 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:44.376738 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:44.377757 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:44.377766 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:44.379054 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:44.379058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:44.379209 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:44.379270 (+ 61us) client_negotiation.cc:769] Sending connection context
0810 22:33:44.379343 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 22:33:44.379437 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":894,"mutex_wait_us":46,"thread_start_us":120,"threads_started":1}
I20260810 22:33:44.380065 19106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:44.373917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36986 (local address 127.16.233.67:43165)
0810 22:33:44.374356 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:44.374367 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:44.374796 (+ 429us) server_negotiation.cc:407] Connection header received
0810 22:33:44.374953 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:44.374958 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:44.375127 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:44.375274 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:33:44.375515 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:44.376607 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:44.377911 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:44.378724 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:44.378837 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:44.378998 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:44.379772 (+ 774us) server_negotiation.cc:299] Negotiation successful
0810 22:33:44.379885 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":125,"threads_started":1}
I20260810 22:33:44.380546 19107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:44.374162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50092 (local address 127.16.233.65:35253)
0810 22:33:44.374579 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:44.374585 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:44.374607 (+ 22us) server_negotiation.cc:407] Connection header received
0810 22:33:44.374680 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:44.374687 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:44.374880 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:44.375045 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:33:44.375322 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:44.376092 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:44.377947 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:44.378724 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:44.378837 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:44.378983 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:44.380292 (+ 1309us) server_negotiation.cc:299] Negotiation successful
0810 22:33:44.380420 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":112,"threads_started":1}
I20260810 22:33:44.380963 19105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:44.373364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.65:35253 (local address 127.0.0.1:50092)
0810 22:33:44.374397 (+ 1033us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:44.374414 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:44.374500 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:44.375024 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:44.375027 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:44.375046 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:33:44.375161 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:44.375167 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:44.376219 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:44.376226 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:44.377846 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:44.377857 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:44.379054 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:44.379058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:44.379211 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:44.379273 (+ 62us) client_negotiation.cc:769] Sending connection context
0810 22:33:44.379344 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 22:33:44.380838 (+ 1494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":926,"mutex_wait_us":1662,"thread_start_us":105,"threads_started":1}
I20260810 22:33:44.382936 18855 tablet_service.cc:1467] Processing CreateTablet for tablet 5174121063d24b95a1bb41802e19d4e0 (DEFAULT_TABLE table=test-table [id=1e376528b97a45fc9204df41751b7fed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:33:44.382933 19012 tablet_service.cc:1467] Processing CreateTablet for tablet 5174121063d24b95a1bb41802e19d4e0 (DEFAULT_TABLE table=test-table [id=1e376528b97a45fc9204df41751b7fed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:33:44.383210 18703 tablet_service.cc:1467] Processing CreateTablet for tablet 5174121063d24b95a1bb41802e19d4e0 (DEFAULT_TABLE table=test-table [id=1e376528b97a45fc9204df41751b7fed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:33:44.384135 18855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5174121063d24b95a1bb41802e19d4e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:33:44.384205 18703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5174121063d24b95a1bb41802e19d4e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:33:44.384140 19012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5174121063d24b95a1bb41802e19d4e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:33:44.390623 19108 tablet_bootstrap.cc:492] T 5174121063d24b95a1bb41802e19d4e0 P c1914598e5c74226b1200a6f2c694c8f: Bootstrap starting.
I20260810 22:33:44.392213 19110 tablet_bootstrap.cc:492] T 5174121063d24b95a1bb41802e19d4e0 P d4f02cff8e1a4d8f98e81b5b2e80becc: Bootstrap starting.
I20260810 22:33:44.393226 19109 tablet_bootstrap.cc:492] T 5174121063d24b95a1bb41802e19d4e0 P 83c6dfaf96b34764967f6f5497309bf1: Bootstrap starting.
I20260810 22:33:44.393244 19108 tablet_bootstrap.cc:654] T 5174121063d24b95a1bb41802e19d4e0 P c1914598e5c74226b1200a6f2c694c8f: Neither blocks nor log segments found. Creating new log.
I20260810 22:33:44.394131 19108 log.cc:824] T 5174121063d24b95a1bb41802e19d4e0 P c1914598e5c74226b1200a6f2c694c8f: Log is configured to *not* fsync() on all Append() calls
I20260810 22:33:44.394418 19110 tablet_bootstrap.cc:654] T 5174121063d24b95a1bb41802e19d4e0 P d4f02cff8e1a4d8f98e81b5b2e80becc: Neither blocks nor log segments found. Creating new log.
I20260810 22:33:44.395177 19109 tablet_bootstrap.cc:654] T 5174121063d24b95a1bb41802e19d4e0 P 83c6dfaf96b34764967f6f5497309bf1: Neither blocks nor log segments found. Creating new log.
I20260810 22:33:44.395236 19110 log.cc:824] T 5174121063d24b95a1bb41802e19d4e0 P d4f02cff8e1a4d8f98e81b5b2e80becc: Log is configured to *not* fsync() on all Append() calls
I20260810 22:33:44.395910 19109 log.cc:824] T 5174121063d24b95a1bb41802e19d4e0 P 83c6dfaf96b34764967f6f5497309bf1: Log is configured to *not* fsync() on all Append() calls
I20260810 22:33:44.396567 19108 tablet_bootstrap.cc:492] T 5174121063d24b95a1bb41802e19d4e0 P c1914598e5c74226b1200a6f2c694c8f: No bootstrap required, opened a new log
I20260810 22:33:44.396787 19108 ts_tablet_manager.cc:1397] T 5174121063d24b95a1bb41802e19d4e0 P c1914598e5c74226b1200a6f2c694c8f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 22:33:44.397253 19110 tablet_bootstrap.cc:492] T 5174121063d24b95a1bb41802e19d4e0 P d4f02cff8e1a4d8f98e81b5b2e80becc: No bootstrap required, opened a new log
I20260810 22:33:44.397441 19110 ts_tablet_manager.cc:1397] T 5174121063d24b95a1bb41802e19d4e0 P d4f02cff8e1a4d8f98e81b5b2e80becc: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 22:33:44.397843 19109 tablet_bootstrap.cc:492] T 5174121063d24b95a1bb41802e19d4e0 P 83c6dfaf96b34764967f6f5497309bf1: No bootstrap required, opened a new log
I20260810 22:33:44.398046 19109 ts_tablet_manager.cc:1397] T 5174121063d24b95a1bb41802e19d4e0 P 83c6dfaf96b34764967f6f5497309bf1: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260810 22:33:44.401098 19109 raft_consensus.cc:348] T 5174121063d24b95a1bb41802e19d4e0 P 83c6dfaf96b34764967f6f5497309bf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4f02cff8e1a4d8f98e81b5b2e80becc" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39683 } } peers { permanent_uuid: "83c6dfaf96b34764967f6f5497309bf1" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 43165 } } peers { permanent_uuid: "c1914598e5c74226b1200a6f2c694c8f" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 35253 } }
I20260810 22:33:44.401440 19109 raft_consensus.cc:374] T 5174121063d24b95a1bb41802e19d4e0 P 83c6dfaf96b34764967f6f5497309bf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:33:44.401499 19109 raft_consensus.cc:729] T 5174121063d24b95a1bb41802e19d4e0 P 83c6dfaf96b34764967f6f5497309bf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83c6dfaf96b34764967f6f5497309bf1, State: Initialized, Role: FOLLOWER
I20260810 22:33:44.401665 19110 raft_consensus.cc:348] T 5174121063d24b95a1bb41802e19d4e0 P d4f02cff8e1a4d8f98e81b5b2e80becc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4f02cff8e1a4d8f98e81b5b2e80becc" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39683 } } peers { permanent_uuid: "83c6dfaf96b34764967f6f5497309bf1" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 43165 } } peers { permanent_uuid: "c1914598e5c74226b1200a6f2c694c8f" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 35253 } }
I20260810 22:33:44.401665 19108 raft_consensus.cc:348] T 5174121063d24b95a1bb41802e19d4e0 P c1914598e5c74226b1200a6f2c694c8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4f02cff8e1a4d8f98e81b5b2e80becc" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39683 } } peers { permanent_uuid: "83c6dfaf96b34764967f6f5497309bf1" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 43165 } } peers { permanent_uuid: "c1914598e5c74226b1200a6f2c694c8f" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 35253 } }
I20260810 22:33:44.401911 19108 raft_consensus.cc:374] T 5174121063d24b95a1bb41802e19d4e0 P c1914598e5c74226b1200a6f2c694c8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:33:44.401911 19110 raft_consensus.cc:374] T 5174121063d24b95a1bb41802e19d4e0 P d4f02cff8e1a4d8f98e81b5b2e80becc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:33:44.401961 19110 raft_consensus.cc:729] T 5174121063d24b95a1bb41802e19d4e0 P d4f02cff8e1a4d8f98e81b5b2e80becc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4f02cff8e1a4d8f98e81b5b2e80becc, State: Initialized, Role: FOLLOWER
I20260810 22:33:44.401961 19108 raft_consensus.cc:729] T 5174121063d24b95a1bb41802e19d4e0 P c1914598e5c74226b1200a6f2c694c8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1914598e5c74226b1200a6f2c694c8f, State: Initialized, Role: FOLLOWER
I20260810 22:33:44.402000 19109 consensus_queue.cc:260] T 5174121063d24b95a1bb41802e19d4e0 P 83c6dfaf96b34764967f6f5497309bf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4f02cff8e1a4d8f98e81b5b2e80becc" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39683 } } peers { permanent_uuid: "83c6dfaf96b34764967f6f5497309bf1" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 43165 } } peers { permanent_uuid: "c1914598e5c74226b1200a6f2c694c8f" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 35253 } }
I20260810 22:33:44.402380 19110 consensus_queue.cc:260] T 5174121063d24b95a1bb41802e19d4e0 P d4f02cff8e1a4d8f98e81b5b2e80becc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4f02cff8e1a4d8f98e81b5b2e80becc" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39683 } } peers { permanent_uuid: "83c6dfaf96b34764967f6f5497309bf1" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 43165 } } peers { permanent_uuid: "c1914598e5c74226b1200a6f2c694c8f" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 35253 } }
I20260810 22:33:44.402377 19108 consensus_queue.cc:260] T 5174121063d24b95a1bb41802e19d4e0 P c1914598e5c74226b1200a6f2c694c8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4f02cff8e1a4d8f98e81b5b2e80becc" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39683 } } peers { permanent_uuid: "83c6dfaf96b34764967f6f5497309bf1" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 43165 } } peers { permanent_uuid: "c1914598e5c74226b1200a6f2c694c8f" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 35253 } }
I20260810 22:33:44.403175 19109 ts_tablet_manager.cc:1428] T 5174121063d24b95a1bb41802e19d4e0 P 83c6dfaf96b34764967f6f5497309bf1: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20260810 22:33:44.403309 19108 ts_tablet_manager.cc:1428] T 5174121063d24b95a1bb41802e19d4e0 P c1914598e5c74226b1200a6f2c694c8f: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260810 22:33:44.403309 19110 ts_tablet_manager.cc:1428] T 5174121063d24b95a1bb41802e19d4e0 P d4f02cff8e1a4d8f98e81b5b2e80becc: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20260810 22:33:44.403592 19083 heartbeater.cc:503] Master 127.16.233.126:43649 was elected leader, sending a full tablet report...
I20260810 22:33:44.403650 18774 heartbeater.cc:503] Master 127.16.233.126:43649 was elected leader, sending a full tablet report...
I20260810 22:33:44.403650 18927 heartbeater.cc:503] Master 127.16.233.126:43649 was elected leader, sending a full tablet report...
I20260810 22:33:44.545948 19115 raft_consensus.cc:482] T 5174121063d24b95a1bb41802e19d4e0 P c1914598e5c74226b1200a6f2c694c8f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:33:44.546213 19115 raft_consensus.cc:504] T 5174121063d24b95a1bb41802e19d4e0 P c1914598e5c74226b1200a6f2c694c8f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4f02cff8e1a4d8f98e81b5b2e80becc" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39683 } } peers { permanent_uuid: "83c6dfaf96b34764967f6f5497309bf1" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 43165 } } peers { permanent_uuid: "c1914598e5c74226b1200a6f2c694c8f" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 35253 } }
I20260810 22:33:44.547549 19115 leader_election.cc:288] T 5174121063d24b95a1bb41802e19d4e0 P c1914598e5c74226b1200a6f2c694c8f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4f02cff8e1a4d8f98e81b5b2e80becc (127.16.233.66:39683), 83c6dfaf96b34764967f6f5497309bf1 (127.16.233.67:43165)
I20260810 22:33:44.552497 19103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:44.547969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:50673 (local address 127.16.233.66:39683)
0810 22:33:44.548253 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:44.548258 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:44.548465 (+ 207us) server_negotiation.cc:407] Connection header received
0810 22:33:44.548693 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:44.548696 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:44.548764 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:44.548883 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:33:44.549212 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:44.549848 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:44.551077 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:44.551715 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:44.551817 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:44.551857 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:44.552218 (+ 361us) server_negotiation.cc:299] Negotiation successful
0810 22:33:44.552347 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174}
I20260810 22:33:44.552718 19117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:44.547780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.66:39683 (local address 127.16.233.65:50673)
0810 22:33:44.548306 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:44.548341 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:44.548496 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:44.548898 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:44.548902 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:44.548924 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:33:44.549081 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:44.549092 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:44.550024 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:44.550028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:44.550922 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:44.550933 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:44.551916 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:44.551919 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:44.552028 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:44.552058 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 22:33:44.552163 (+ 105us) client_negotiation.cc:240] Negotiation successful
0810 22:33:44.552240 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":174,"threads_started":1}
I20260810 22:33:44.553185 19106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:44.548453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:60607 (local address 127.16.233.67:43165)
0810 22:33:44.548592 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:44.548598 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:44.548858 (+ 260us) server_negotiation.cc:407] Connection header received
0810 22:33:44.548983 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:44.548987 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:44.549036 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:44.549126 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:33:44.549434 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:44.550024 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:44.551603 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:44.552255 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:44.552422 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:44.552479 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:44.552966 (+ 487us) server_negotiation.cc:299] Negotiation successful
0810 22:33:44.553055 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 22:33:44.553197 19118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:44.548321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.67:43165 (local address 127.16.233.65:60607)
0810 22:33:44.548705 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:44.548719 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:44.548857 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:44.549172 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:44.549177 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:44.549196 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:33:44.549309 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:44.549316 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:44.550219 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:44.550221 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:44.551438 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:44.551449 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:44.552418 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:44.552423 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:44.552620 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:44.552657 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 22:33:44.552923 (+ 266us) client_negotiation.cc:240] Negotiation successful
0810 22:33:44.553017 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":117,"threads_started":1}
I20260810 22:33:44.553551 18877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5174121063d24b95a1bb41802e19d4e0" candidate_uuid: "c1914598e5c74226b1200a6f2c694c8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4f02cff8e1a4d8f98e81b5b2e80becc" is_pre_election: true
I20260810 22:33:44.553913 18877 raft_consensus.cc:2454] T 5174121063d24b95a1bb41802e19d4e0 P d4f02cff8e1a4d8f98e81b5b2e80becc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1914598e5c74226b1200a6f2c694c8f in term 0.
I20260810 22:33:44.554075 19032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5174121063d24b95a1bb41802e19d4e0" candidate_uuid: "c1914598e5c74226b1200a6f2c694c8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c6dfaf96b34764967f6f5497309bf1" is_pre_election: true
I20260810 22:33:44.554399 19032 raft_consensus.cc:2454] T 5174121063d24b95a1bb41802e19d4e0 P 83c6dfaf96b34764967f6f5497309bf1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1914598e5c74226b1200a6f2c694c8f in term 0.
I20260810 22:33:44.554404 18649 leader_election.cc:302] T 5174121063d24b95a1bb41802e19d4e0 P c1914598e5c74226b1200a6f2c694c8f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1914598e5c74226b1200a6f2c694c8f, d4f02cff8e1a4d8f98e81b5b2e80becc; no voters:
I20260810 22:33:44.554731 19115 raft_consensus.cc:2781] T 5174121063d24b95a1bb41802e19d4e0 P c1914598e5c74226b1200a6f2c694c8f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:33:44.554873 19115 raft_consensus.cc:482] T 5174121063d24b95a1bb41802e19d4e0 P c1914598e5c74226b1200a6f2c694c8f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:33:44.554944 19115 raft_consensus.cc:3037] T 5174121063d24b95a1bb41802e19d4e0 P c1914598e5c74226b1200a6f2c694c8f [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:33:44.556510 19115 raft_consensus.cc:504] T 5174121063d24b95a1bb41802e19d4e0 P c1914598e5c74226b1200a6f2c694c8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4f02cff8e1a4d8f98e81b5b2e80becc" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39683 } } peers { permanent_uuid: "83c6dfaf96b34764967f6f5497309bf1" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 43165 } } peers { permanent_uuid: "c1914598e5c74226b1200a6f2c694c8f" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 35253 } }
I20260810 22:33:44.556941 19115 leader_election.cc:288] T 5174121063d24b95a1bb41802e19d4e0 P c1914598e5c74226b1200a6f2c694c8f [CANDIDATE]: Term 1 election: Requested vote from peers d4f02cff8e1a4d8f98e81b5b2e80becc (127.16.233.66:39683), 83c6dfaf96b34764967f6f5497309bf1 (127.16.233.67:43165)
I20260810 22:33:44.557308 18877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5174121063d24b95a1bb41802e19d4e0" candidate_uuid: "c1914598e5c74226b1200a6f2c694c8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4f02cff8e1a4d8f98e81b5b2e80becc"
I20260810 22:33:44.557469 19032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5174121063d24b95a1bb41802e19d4e0" candidate_uuid: "c1914598e5c74226b1200a6f2c694c8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c6dfaf96b34764967f6f5497309bf1"
I20260810 22:33:44.557519 18877 raft_consensus.cc:3037] T 5174121063d24b95a1bb41802e19d4e0 P d4f02cff8e1a4d8f98e81b5b2e80becc [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:33:44.557585 19032 raft_consensus.cc:3037] T 5174121063d24b95a1bb41802e19d4e0 P 83c6dfaf96b34764967f6f5497309bf1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:33:44.559213 18877 raft_consensus.cc:2454] T 5174121063d24b95a1bb41802e19d4e0 P d4f02cff8e1a4d8f98e81b5b2e80becc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1914598e5c74226b1200a6f2c694c8f in term 1.
I20260810 22:33:44.559213 19032 raft_consensus.cc:2454] T 5174121063d24b95a1bb41802e19d4e0 P 83c6dfaf96b34764967f6f5497309bf1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1914598e5c74226b1200a6f2c694c8f in term 1.
I20260810 22:33:44.559537 18649 leader_election.cc:302] T 5174121063d24b95a1bb41802e19d4e0 P c1914598e5c74226b1200a6f2c694c8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1914598e5c74226b1200a6f2c694c8f, d4f02cff8e1a4d8f98e81b5b2e80becc; no voters:
I20260810 22:33:44.559759 19115 raft_consensus.cc:2781] T 5174121063d24b95a1bb41802e19d4e0 P c1914598e5c74226b1200a6f2c694c8f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:33:44.560082 19115 raft_consensus.cc:686] T 5174121063d24b95a1bb41802e19d4e0 P c1914598e5c74226b1200a6f2c694c8f [term 1 LEADER]: Becoming Leader. State: Replica: c1914598e5c74226b1200a6f2c694c8f, State: Running, Role: LEADER
I20260810 22:33:44.560521 19115 consensus_queue.cc:237] T 5174121063d24b95a1bb41802e19d4e0 P c1914598e5c74226b1200a6f2c694c8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4f02cff8e1a4d8f98e81b5b2e80becc" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39683 } } peers { permanent_uuid: "83c6dfaf96b34764967f6f5497309bf1" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 43165 } } peers { permanent_uuid: "c1914598e5c74226b1200a6f2c694c8f" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 35253 } }
W20260810 22:33:44.564574 19084 tablet.cc:2369] T 5174121063d24b95a1bb41802e19d4e0 P 83c6dfaf96b34764967f6f5497309bf1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:33:44.564606 18547 catalog_manager.cc:5495] T 5174121063d24b95a1bb41802e19d4e0 P c1914598e5c74226b1200a6f2c694c8f reported cstate change: term changed from 0 to 1, leader changed from <none> to c1914598e5c74226b1200a6f2c694c8f (127.16.233.65). New cstate: current_term: 1 leader_uuid: "c1914598e5c74226b1200a6f2c694c8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4f02cff8e1a4d8f98e81b5b2e80becc" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83c6dfaf96b34764967f6f5497309bf1" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 43165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1914598e5c74226b1200a6f2c694c8f" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 35253 } health_report { overall_health: HEALTHY } } }
W20260810 22:33:44.566470 18775 tablet.cc:2369] T 5174121063d24b95a1bb41802e19d4e0 P c1914598e5c74226b1200a6f2c694c8f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 22:33:44.569350 18928 tablet.cc:2369] T 5174121063d24b95a1bb41802e19d4e0 P d4f02cff8e1a4d8f98e81b5b2e80becc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:33:44.627244 19107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:44.623068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50100 (local address 127.16.233.65:35253)
0810 22:33:44.623213 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:44.623216 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:44.623233 (+ 17us) server_negotiation.cc:407] Connection header received
0810 22:33:44.623380 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:44.623382 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:44.623444 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:44.623590 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:33:44.623903 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:44.624476 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:44.625474 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:44.625711 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:44.625808 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:44.626208 (+ 400us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:33:44.626703 (+ 495us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:33:44.626875 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:44.627047 (+ 172us) server_negotiation.cc:299] Negotiation successful
0810 22:33:44.627112 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 22:33:44.635357 18877 raft_consensus.cc:1261] T 5174121063d24b95a1bb41802e19d4e0 P d4f02cff8e1a4d8f98e81b5b2e80becc [term 1 FOLLOWER]: Refusing update from remote peer c1914598e5c74226b1200a6f2c694c8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:33:44.636240 19119 consensus_queue.cc:1035] T 5174121063d24b95a1bb41802e19d4e0 P c1914598e5c74226b1200a6f2c694c8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4f02cff8e1a4d8f98e81b5b2e80becc" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:33:44.637151 19032 raft_consensus.cc:1261] T 5174121063d24b95a1bb41802e19d4e0 P 83c6dfaf96b34764967f6f5497309bf1 [term 1 FOLLOWER]: Refusing update from remote peer c1914598e5c74226b1200a6f2c694c8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:33:44.637905 19119 consensus_queue.cc:1035] T 5174121063d24b95a1bb41802e19d4e0 P c1914598e5c74226b1200a6f2c694c8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "83c6dfaf96b34764967f6f5497309bf1" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 43165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:33:44.645459 19123 mvcc.cc:205] Tried to move back new op lower bound from 7317099416096792576 to 7317099415803174912. Current Snapshot: MvccSnapshot[applied={T|T < 7317099416096792576}]
I20260810 22:33:44.649209 19125 mvcc.cc:205] Tried to move back new op lower bound from 7317099416096792576 to 7317099415803174912. Current Snapshot: MvccSnapshot[applied={T|T < 7317099416096792576}]
I20260810 22:33:44.660444 18548 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:33:44.660778 18548 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:33:44.664409 18548 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 04121952a88449349c980197e530ae20: Sending DeleteTablet for 3 replicas of tablet 5174121063d24b95a1bb41802e19d4e0
I20260810 22:33:44.665582 18703 tablet_service.cc:1514] Processing DeleteTablet for tablet 5174121063d24b95a1bb41802e19d4e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:33:44 UTC) from {username='kudu', principal='kudu/127.16.233.126@KRBTEST.COM'} at 127.0.0.1:50092
I20260810 22:33:44.665575 18855 tablet_service.cc:1514] Processing DeleteTablet for tablet 5174121063d24b95a1bb41802e19d4e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:33:44 UTC) from {username='kudu', principal='kudu/127.16.233.126@KRBTEST.COM'} at 127.0.0.1:40080
I20260810 22:33:44.665719 19012 tablet_service.cc:1514] Processing DeleteTablet for tablet 5174121063d24b95a1bb41802e19d4e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:33:44 UTC) from {username='kudu', principal='kudu/127.16.233.126@KRBTEST.COM'} at 127.0.0.1:36986
I20260810 22:33:44.666369 19133 tablet_replica.cc:330] T 5174121063d24b95a1bb41802e19d4e0 P c1914598e5c74226b1200a6f2c694c8f: stopping tablet replica
I20260810 22:33:44.666525 19132 tablet_replica.cc:330] T 5174121063d24b95a1bb41802e19d4e0 P d4f02cff8e1a4d8f98e81b5b2e80becc: stopping tablet replica
I20260810 22:33:44.666815 19133 raft_consensus.cc:2229] T 5174121063d24b95a1bb41802e19d4e0 P c1914598e5c74226b1200a6f2c694c8f [term 1 LEADER]: Raft consensus shutting down.
I20260810 22:33:44.666918 19132 raft_consensus.cc:2229] T 5174121063d24b95a1bb41802e19d4e0 P d4f02cff8e1a4d8f98e81b5b2e80becc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:33:44.667272 19133 raft_consensus.cc:2258] T 5174121063d24b95a1bb41802e19d4e0 P c1914598e5c74226b1200a6f2c694c8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:33:44.667358 19132 raft_consensus.cc:2258] T 5174121063d24b95a1bb41802e19d4e0 P d4f02cff8e1a4d8f98e81b5b2e80becc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:33:44.667321 18548 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40372:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 22:33:44.667763 19134 tablet_replica.cc:330] T 5174121063d24b95a1bb41802e19d4e0 P 83c6dfaf96b34764967f6f5497309bf1: stopping tablet replica
W20260810 22:33:44.667819 18548 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:33:44.668107 19134 raft_consensus.cc:2229] T 5174121063d24b95a1bb41802e19d4e0 P 83c6dfaf96b34764967f6f5497309bf1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:33:44.668421 19134 raft_consensus.cc:2258] T 5174121063d24b95a1bb41802e19d4e0 P 83c6dfaf96b34764967f6f5497309bf1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:33:44.669106 19133 ts_tablet_manager.cc:1905] T 5174121063d24b95a1bb41802e19d4e0 P c1914598e5c74226b1200a6f2c694c8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:33:44.670158 19132 ts_tablet_manager.cc:1905] T 5174121063d24b95a1bb41802e19d4e0 P d4f02cff8e1a4d8f98e81b5b2e80becc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:33:44.673466 19132 ts_tablet_manager.cc:1918] T 5174121063d24b95a1bb41802e19d4e0 P d4f02cff8e1a4d8f98e81b5b2e80becc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:33:44.673599 19132 log.cc:1196] T 5174121063d24b95a1bb41802e19d4e0 P d4f02cff8e1a4d8f98e81b5b2e80becc: Deleting WAL directory at /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/ts-1/wal/wals/5174121063d24b95a1bb41802e19d4e0
I20260810 22:33:44.673961 19132 ts_tablet_manager.cc:1939] T 5174121063d24b95a1bb41802e19d4e0 P d4f02cff8e1a4d8f98e81b5b2e80becc: Deleting consensus metadata
I20260810 22:33:44.674384 19133 ts_tablet_manager.cc:1918] T 5174121063d24b95a1bb41802e19d4e0 P c1914598e5c74226b1200a6f2c694c8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:33:44.674513 19133 log.cc:1196] T 5174121063d24b95a1bb41802e19d4e0 P c1914598e5c74226b1200a6f2c694c8f: Deleting WAL directory at /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/ts-0/wal/wals/5174121063d24b95a1bb41802e19d4e0
I20260810 22:33:44.674854 19133 ts_tablet_manager.cc:1939] T 5174121063d24b95a1bb41802e19d4e0 P c1914598e5c74226b1200a6f2c694c8f: Deleting consensus metadata
I20260810 22:33:44.675417 18536 catalog_manager.cc:4841] TS d4f02cff8e1a4d8f98e81b5b2e80becc (127.16.233.66:39683): tablet 5174121063d24b95a1bb41802e19d4e0 (table test-table [id=1e376528b97a45fc9204df41751b7fed]) successfully deleted
I20260810 22:33:44.676100 18537 catalog_manager.cc:4841] TS c1914598e5c74226b1200a6f2c694c8f (127.16.233.65:35253): tablet 5174121063d24b95a1bb41802e19d4e0 (table test-table [id=1e376528b97a45fc9204df41751b7fed]) successfully deleted
I20260810 22:33:44.677043 19134 ts_tablet_manager.cc:1905] T 5174121063d24b95a1bb41802e19d4e0 P 83c6dfaf96b34764967f6f5497309bf1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:33:44.677935 18703 tablet_service.cc:1467] Processing CreateTablet for tablet b22fdf1a39cf4f098b3a987e9bae6d83 (DEFAULT_TABLE table=test-table [id=df1c088b750544a3af78a988aa7dc433]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:33:44.678247 18703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b22fdf1a39cf4f098b3a987e9bae6d83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:33:44.678369 18855 tablet_service.cc:1467] Processing CreateTablet for tablet b22fdf1a39cf4f098b3a987e9bae6d83 (DEFAULT_TABLE table=test-table [id=df1c088b750544a3af78a988aa7dc433]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:33:44.678642 18855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b22fdf1a39cf4f098b3a987e9bae6d83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:33:44.680207 19134 ts_tablet_manager.cc:1918] T 5174121063d24b95a1bb41802e19d4e0 P 83c6dfaf96b34764967f6f5497309bf1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:33:44.680325 19134 log.cc:1196] T 5174121063d24b95a1bb41802e19d4e0 P 83c6dfaf96b34764967f6f5497309bf1: Deleting WAL directory at /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/ts-2/wal/wals/5174121063d24b95a1bb41802e19d4e0
I20260810 22:33:44.680713 19134 ts_tablet_manager.cc:1939] T 5174121063d24b95a1bb41802e19d4e0 P 83c6dfaf96b34764967f6f5497309bf1: Deleting consensus metadata
I20260810 22:33:44.681444 19108 tablet_bootstrap.cc:492] T b22fdf1a39cf4f098b3a987e9bae6d83 P c1914598e5c74226b1200a6f2c694c8f: Bootstrap starting.
I20260810 22:33:44.681540 19012 tablet_service.cc:1467] Processing CreateTablet for tablet b22fdf1a39cf4f098b3a987e9bae6d83 (DEFAULT_TABLE table=test-table [id=df1c088b750544a3af78a988aa7dc433]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:33:44.681800 19012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b22fdf1a39cf4f098b3a987e9bae6d83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:33:44.682816 19110 tablet_bootstrap.cc:492] T b22fdf1a39cf4f098b3a987e9bae6d83 P d4f02cff8e1a4d8f98e81b5b2e80becc: Bootstrap starting.
I20260810 22:33:44.683272 19108 tablet_bootstrap.cc:654] T b22fdf1a39cf4f098b3a987e9bae6d83 P c1914598e5c74226b1200a6f2c694c8f: Neither blocks nor log segments found. Creating new log.
I20260810 22:33:44.684255 19110 tablet_bootstrap.cc:654] T b22fdf1a39cf4f098b3a987e9bae6d83 P d4f02cff8e1a4d8f98e81b5b2e80becc: Neither blocks nor log segments found. Creating new log.
I20260810 22:33:44.685318 19109 tablet_bootstrap.cc:492] T b22fdf1a39cf4f098b3a987e9bae6d83 P 83c6dfaf96b34764967f6f5497309bf1: Bootstrap starting.
I20260810 22:33:44.685848 18536 catalog_manager.cc:4841] TS 83c6dfaf96b34764967f6f5497309bf1 (127.16.233.67:43165): tablet 5174121063d24b95a1bb41802e19d4e0 (table test-table [id=1e376528b97a45fc9204df41751b7fed]) successfully deleted
I20260810 22:33:44.686744 19108 tablet_bootstrap.cc:492] T b22fdf1a39cf4f098b3a987e9bae6d83 P c1914598e5c74226b1200a6f2c694c8f: No bootstrap required, opened a new log
I20260810 22:33:44.686877 19108 ts_tablet_manager.cc:1397] T b22fdf1a39cf4f098b3a987e9bae6d83 P c1914598e5c74226b1200a6f2c694c8f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260810 22:33:44.687341 19109 tablet_bootstrap.cc:654] T b22fdf1a39cf4f098b3a987e9bae6d83 P 83c6dfaf96b34764967f6f5497309bf1: Neither blocks nor log segments found. Creating new log.
I20260810 22:33:44.687888 19108 raft_consensus.cc:348] T b22fdf1a39cf4f098b3a987e9bae6d83 P c1914598e5c74226b1200a6f2c694c8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1914598e5c74226b1200a6f2c694c8f" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 35253 } } peers { permanent_uuid: "d4f02cff8e1a4d8f98e81b5b2e80becc" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39683 } } peers { permanent_uuid: "83c6dfaf96b34764967f6f5497309bf1" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 43165 } }
I20260810 22:33:44.688050 19108 raft_consensus.cc:374] T b22fdf1a39cf4f098b3a987e9bae6d83 P c1914598e5c74226b1200a6f2c694c8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:33:44.688110 19108 raft_consensus.cc:729] T b22fdf1a39cf4f098b3a987e9bae6d83 P c1914598e5c74226b1200a6f2c694c8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1914598e5c74226b1200a6f2c694c8f, State: Initialized, Role: FOLLOWER
I20260810 22:33:44.688279 19108 consensus_queue.cc:260] T b22fdf1a39cf4f098b3a987e9bae6d83 P c1914598e5c74226b1200a6f2c694c8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1914598e5c74226b1200a6f2c694c8f" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 35253 } } peers { permanent_uuid: "d4f02cff8e1a4d8f98e81b5b2e80becc" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39683 } } peers { permanent_uuid: "83c6dfaf96b34764967f6f5497309bf1" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 43165 } }
I20260810 22:33:44.688668 19108 ts_tablet_manager.cc:1428] T b22fdf1a39cf4f098b3a987e9bae6d83 P c1914598e5c74226b1200a6f2c694c8f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260810 22:33:44.689270 19109 tablet_bootstrap.cc:492] T b22fdf1a39cf4f098b3a987e9bae6d83 P 83c6dfaf96b34764967f6f5497309bf1: No bootstrap required, opened a new log
I20260810 22:33:44.689371 19109 ts_tablet_manager.cc:1397] T b22fdf1a39cf4f098b3a987e9bae6d83 P 83c6dfaf96b34764967f6f5497309bf1: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.002s
I20260810 22:33:44.689563 19110 tablet_bootstrap.cc:492] T b22fdf1a39cf4f098b3a987e9bae6d83 P d4f02cff8e1a4d8f98e81b5b2e80becc: No bootstrap required, opened a new log
I20260810 22:33:44.689651 19110 ts_tablet_manager.cc:1397] T b22fdf1a39cf4f098b3a987e9bae6d83 P d4f02cff8e1a4d8f98e81b5b2e80becc: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260810 22:33:44.690160 19109 raft_consensus.cc:348] T b22fdf1a39cf4f098b3a987e9bae6d83 P 83c6dfaf96b34764967f6f5497309bf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1914598e5c74226b1200a6f2c694c8f" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 35253 } } peers { permanent_uuid: "d4f02cff8e1a4d8f98e81b5b2e80becc" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39683 } } peers { permanent_uuid: "83c6dfaf96b34764967f6f5497309bf1" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 43165 } }
I20260810 22:33:44.690315 19109 raft_consensus.cc:374] T b22fdf1a39cf4f098b3a987e9bae6d83 P 83c6dfaf96b34764967f6f5497309bf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:33:44.690351 19109 raft_consensus.cc:729] T b22fdf1a39cf4f098b3a987e9bae6d83 P 83c6dfaf96b34764967f6f5497309bf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83c6dfaf96b34764967f6f5497309bf1, State: Initialized, Role: FOLLOWER
I20260810 22:33:44.690372 19110 raft_consensus.cc:348] T b22fdf1a39cf4f098b3a987e9bae6d83 P d4f02cff8e1a4d8f98e81b5b2e80becc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1914598e5c74226b1200a6f2c694c8f" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 35253 } } peers { permanent_uuid: "d4f02cff8e1a4d8f98e81b5b2e80becc" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39683 } } peers { permanent_uuid: "83c6dfaf96b34764967f6f5497309bf1" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 43165 } }
I20260810 22:33:44.690490 19110 raft_consensus.cc:374] T b22fdf1a39cf4f098b3a987e9bae6d83 P d4f02cff8e1a4d8f98e81b5b2e80becc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:33:44.690521 19110 raft_consensus.cc:729] T b22fdf1a39cf4f098b3a987e9bae6d83 P d4f02cff8e1a4d8f98e81b5b2e80becc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4f02cff8e1a4d8f98e81b5b2e80becc, State: Initialized, Role: FOLLOWER
I20260810 22:33:44.690490 19109 consensus_queue.cc:260] T b22fdf1a39cf4f098b3a987e9bae6d83 P 83c6dfaf96b34764967f6f5497309bf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1914598e5c74226b1200a6f2c694c8f" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 35253 } } peers { permanent_uuid: "d4f02cff8e1a4d8f98e81b5b2e80becc" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39683 } } peers { permanent_uuid: "83c6dfaf96b34764967f6f5497309bf1" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 43165 } }
I20260810 22:33:44.690682 19110 consensus_queue.cc:260] T b22fdf1a39cf4f098b3a987e9bae6d83 P d4f02cff8e1a4d8f98e81b5b2e80becc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1914598e5c74226b1200a6f2c694c8f" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 35253 } } peers { permanent_uuid: "d4f02cff8e1a4d8f98e81b5b2e80becc" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39683 } } peers { permanent_uuid: "83c6dfaf96b34764967f6f5497309bf1" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 43165 } }
I20260810 22:33:44.690820 19109 ts_tablet_manager.cc:1428] T b22fdf1a39cf4f098b3a987e9bae6d83 P 83c6dfaf96b34764967f6f5497309bf1: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260810 22:33:44.690944 19110 ts_tablet_manager.cc:1428] T b22fdf1a39cf4f098b3a987e9bae6d83 P d4f02cff8e1a4d8f98e81b5b2e80becc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 22:33:44.720130 19115 raft_consensus.cc:482] T b22fdf1a39cf4f098b3a987e9bae6d83 P c1914598e5c74226b1200a6f2c694c8f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:33:44.720292 19115 raft_consensus.cc:504] T b22fdf1a39cf4f098b3a987e9bae6d83 P c1914598e5c74226b1200a6f2c694c8f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1914598e5c74226b1200a6f2c694c8f" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 35253 } } peers { permanent_uuid: "d4f02cff8e1a4d8f98e81b5b2e80becc" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39683 } } peers { permanent_uuid: "83c6dfaf96b34764967f6f5497309bf1" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 43165 } }
I20260810 22:33:44.721115 19115 leader_election.cc:288] T b22fdf1a39cf4f098b3a987e9bae6d83 P c1914598e5c74226b1200a6f2c694c8f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4f02cff8e1a4d8f98e81b5b2e80becc (127.16.233.66:39683), 83c6dfaf96b34764967f6f5497309bf1 (127.16.233.67:43165)
I20260810 22:33:44.721750 18877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b22fdf1a39cf4f098b3a987e9bae6d83" candidate_uuid: "c1914598e5c74226b1200a6f2c694c8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4f02cff8e1a4d8f98e81b5b2e80becc" is_pre_election: true
I20260810 22:33:44.721752 19032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b22fdf1a39cf4f098b3a987e9bae6d83" candidate_uuid: "c1914598e5c74226b1200a6f2c694c8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c6dfaf96b34764967f6f5497309bf1" is_pre_election: true
I20260810 22:33:44.721930 19032 raft_consensus.cc:2454] T b22fdf1a39cf4f098b3a987e9bae6d83 P 83c6dfaf96b34764967f6f5497309bf1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1914598e5c74226b1200a6f2c694c8f in term 0.
I20260810 22:33:44.721932 18877 raft_consensus.cc:2454] T b22fdf1a39cf4f098b3a987e9bae6d83 P d4f02cff8e1a4d8f98e81b5b2e80becc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1914598e5c74226b1200a6f2c694c8f in term 0.
I20260810 22:33:44.722280 18649 leader_election.cc:302] T b22fdf1a39cf4f098b3a987e9bae6d83 P c1914598e5c74226b1200a6f2c694c8f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83c6dfaf96b34764967f6f5497309bf1, c1914598e5c74226b1200a6f2c694c8f; no voters:
I20260810 22:33:44.722543 19115 raft_consensus.cc:2781] T b22fdf1a39cf4f098b3a987e9bae6d83 P c1914598e5c74226b1200a6f2c694c8f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:33:44.722661 19115 raft_consensus.cc:482] T b22fdf1a39cf4f098b3a987e9bae6d83 P c1914598e5c74226b1200a6f2c694c8f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:33:44.722743 19115 raft_consensus.cc:3037] T b22fdf1a39cf4f098b3a987e9bae6d83 P c1914598e5c74226b1200a6f2c694c8f [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:33:44.724375 19115 raft_consensus.cc:504] T b22fdf1a39cf4f098b3a987e9bae6d83 P c1914598e5c74226b1200a6f2c694c8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1914598e5c74226b1200a6f2c694c8f" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 35253 } } peers { permanent_uuid: "d4f02cff8e1a4d8f98e81b5b2e80becc" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39683 } } peers { permanent_uuid: "83c6dfaf96b34764967f6f5497309bf1" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 43165 } }
I20260810 22:33:44.724891 19115 leader_election.cc:288] T b22fdf1a39cf4f098b3a987e9bae6d83 P c1914598e5c74226b1200a6f2c694c8f [CANDIDATE]: Term 1 election: Requested vote from peers d4f02cff8e1a4d8f98e81b5b2e80becc (127.16.233.66:39683), 83c6dfaf96b34764967f6f5497309bf1 (127.16.233.67:43165)
I20260810 22:33:44.725483 19032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b22fdf1a39cf4f098b3a987e9bae6d83" candidate_uuid: "c1914598e5c74226b1200a6f2c694c8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c6dfaf96b34764967f6f5497309bf1"
I20260810 22:33:44.725517 18877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b22fdf1a39cf4f098b3a987e9bae6d83" candidate_uuid: "c1914598e5c74226b1200a6f2c694c8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4f02cff8e1a4d8f98e81b5b2e80becc"
I20260810 22:33:44.725648 18877 raft_consensus.cc:3037] T b22fdf1a39cf4f098b3a987e9bae6d83 P d4f02cff8e1a4d8f98e81b5b2e80becc [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:33:44.725649 19032 raft_consensus.cc:3037] T b22fdf1a39cf4f098b3a987e9bae6d83 P 83c6dfaf96b34764967f6f5497309bf1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:33:44.727240 18877 raft_consensus.cc:2454] T b22fdf1a39cf4f098b3a987e9bae6d83 P d4f02cff8e1a4d8f98e81b5b2e80becc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1914598e5c74226b1200a6f2c694c8f in term 1.
I20260810 22:33:44.727240 19032 raft_consensus.cc:2454] T b22fdf1a39cf4f098b3a987e9bae6d83 P 83c6dfaf96b34764967f6f5497309bf1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1914598e5c74226b1200a6f2c694c8f in term 1.
I20260810 22:33:44.727633 18649 leader_election.cc:302] T b22fdf1a39cf4f098b3a987e9bae6d83 P c1914598e5c74226b1200a6f2c694c8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83c6dfaf96b34764967f6f5497309bf1, c1914598e5c74226b1200a6f2c694c8f; no voters:
I20260810 22:33:44.727917 19115 raft_consensus.cc:2781] T b22fdf1a39cf4f098b3a987e9bae6d83 P c1914598e5c74226b1200a6f2c694c8f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:33:44.728066 19115 raft_consensus.cc:686] T b22fdf1a39cf4f098b3a987e9bae6d83 P c1914598e5c74226b1200a6f2c694c8f [term 1 LEADER]: Becoming Leader. State: Replica: c1914598e5c74226b1200a6f2c694c8f, State: Running, Role: LEADER
I20260810 22:33:44.728228 19115 consensus_queue.cc:237] T b22fdf1a39cf4f098b3a987e9bae6d83 P c1914598e5c74226b1200a6f2c694c8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1914598e5c74226b1200a6f2c694c8f" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 35253 } } peers { permanent_uuid: "d4f02cff8e1a4d8f98e81b5b2e80becc" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39683 } } peers { permanent_uuid: "83c6dfaf96b34764967f6f5497309bf1" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 43165 } }
I20260810 22:33:44.730207 18548 catalog_manager.cc:5495] T b22fdf1a39cf4f098b3a987e9bae6d83 P c1914598e5c74226b1200a6f2c694c8f reported cstate change: term changed from 0 to 1, leader changed from <none> to c1914598e5c74226b1200a6f2c694c8f (127.16.233.65). New cstate: current_term: 1 leader_uuid: "c1914598e5c74226b1200a6f2c694c8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1914598e5c74226b1200a6f2c694c8f" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 35253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4f02cff8e1a4d8f98e81b5b2e80becc" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83c6dfaf96b34764967f6f5497309bf1" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 43165 } health_report { overall_health: UNKNOWN } } }
Aug 10 22:33:44 dist-test-slave-77v9 krb5kdc[18499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401223, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.126@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:33:44.782688 19146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:44.771227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:43649 (local address 127.0.0.1:40384)
0810 22:33:44.771783 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:44.771797 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:44.771922 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:44.772218 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:44.772220 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:44.772693 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:33:44.772798 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:44.772804 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:44.774133 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:44.774135 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:44.774694 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:44.774702 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:44.775369 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:44.775371 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:44.775446 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:44.776192 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:33:44.776205 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:33:44.778253 (+ 2048us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:33:44.780618 (+ 2365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:44.780629 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:44.780635 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:44.781046 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:44.781390 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:44.781393 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:44.781394 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:44.781483 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:44.782041 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:33:44.782061 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:33:44.782186 (+ 125us) client_negotiation.cc:769] Sending connection context
0810 22:33:44.782294 (+ 108us) client_negotiation.cc:240] Negotiation successful
0810 22:33:44.782445 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"mutex_wait_us":1,"thread_start_us":225,"threads_started":1}
I20260810 22:33:44.782744 18606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:44.771323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40384 (local address 127.16.233.126:43649)
0810 22:33:44.771527 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:44.771532 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:44.771905 (+ 373us) server_negotiation.cc:407] Connection header received
0810 22:33:44.772004 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:44.772008 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:44.772068 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:44.772184 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:44.773080 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:44.773995 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:44.774835 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:44.775218 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:44.775358 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:44.778456 (+ 3098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:44.778461 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:44.778465 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:44.778498 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:44.780494 (+ 1996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:44.781166 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:44.781170 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:44.781172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:44.781274 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:44.781600 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:44.781605 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:44.781607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:44.781893 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:44.782025 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:44.782419 (+ 394us) server_negotiation.cc:299] Negotiation successful
0810 22:33:44.782577 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260810 22:33:44.788386 18548 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.16.233.126@KRBTEST.COM'} at 127.0.0.1:40384:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 22:33:44.789286 18548 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:33:44.795174 18703 tablet_service.cc:1467] Processing CreateTablet for tablet a83dd71fe3404a02b7aea124048f3c90 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0e371179d6aa436ba7c5f01695fbdcd2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 22:33:44.795351 18855 tablet_service.cc:1467] Processing CreateTablet for tablet a83dd71fe3404a02b7aea124048f3c90 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0e371179d6aa436ba7c5f01695fbdcd2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 22:33:44.795703 18703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a83dd71fe3404a02b7aea124048f3c90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:33:44.795862 18855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a83dd71fe3404a02b7aea124048f3c90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:33:44.798676 19012 tablet_service.cc:1467] Processing CreateTablet for tablet a83dd71fe3404a02b7aea124048f3c90 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0e371179d6aa436ba7c5f01695fbdcd2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 22:33:44.799186 19012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a83dd71fe3404a02b7aea124048f3c90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:33:44.799939 19110 tablet_bootstrap.cc:492] T a83dd71fe3404a02b7aea124048f3c90 P d4f02cff8e1a4d8f98e81b5b2e80becc: Bootstrap starting.
I20260810 22:33:44.801229 19108 tablet_bootstrap.cc:492] T a83dd71fe3404a02b7aea124048f3c90 P c1914598e5c74226b1200a6f2c694c8f: Bootstrap starting.
I20260810 22:33:44.801813 19110 tablet_bootstrap.cc:654] T a83dd71fe3404a02b7aea124048f3c90 P d4f02cff8e1a4d8f98e81b5b2e80becc: Neither blocks nor log segments found. Creating new log.
I20260810 22:33:44.803462 19108 tablet_bootstrap.cc:654] T a83dd71fe3404a02b7aea124048f3c90 P c1914598e5c74226b1200a6f2c694c8f: Neither blocks nor log segments found. Creating new log.
I20260810 22:33:44.803522 19109 tablet_bootstrap.cc:492] T a83dd71fe3404a02b7aea124048f3c90 P 83c6dfaf96b34764967f6f5497309bf1: Bootstrap starting.
I20260810 22:33:44.803660 19110 tablet_bootstrap.cc:492] T a83dd71fe3404a02b7aea124048f3c90 P d4f02cff8e1a4d8f98e81b5b2e80becc: No bootstrap required, opened a new log
I20260810 22:33:44.803848 19110 ts_tablet_manager.cc:1397] T a83dd71fe3404a02b7aea124048f3c90 P d4f02cff8e1a4d8f98e81b5b2e80becc: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260810 22:33:44.804783 19110 raft_consensus.cc:348] T a83dd71fe3404a02b7aea124048f3c90 P d4f02cff8e1a4d8f98e81b5b2e80becc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1914598e5c74226b1200a6f2c694c8f" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 35253 } } peers { permanent_uuid: "d4f02cff8e1a4d8f98e81b5b2e80becc" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39683 } } peers { permanent_uuid: "83c6dfaf96b34764967f6f5497309bf1" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 43165 } }
I20260810 22:33:44.805029 19110 raft_consensus.cc:374] T a83dd71fe3404a02b7aea124048f3c90 P d4f02cff8e1a4d8f98e81b5b2e80becc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:33:44.805078 19110 raft_consensus.cc:729] T a83dd71fe3404a02b7aea124048f3c90 P d4f02cff8e1a4d8f98e81b5b2e80becc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4f02cff8e1a4d8f98e81b5b2e80becc, State: Initialized, Role: FOLLOWER
I20260810 22:33:44.805320 19110 consensus_queue.cc:260] T a83dd71fe3404a02b7aea124048f3c90 P d4f02cff8e1a4d8f98e81b5b2e80becc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1914598e5c74226b1200a6f2c694c8f" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 35253 } } peers { permanent_uuid: "d4f02cff8e1a4d8f98e81b5b2e80becc" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39683 } } peers { permanent_uuid: "83c6dfaf96b34764967f6f5497309bf1" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 43165 } }
I20260810 22:33:44.805652 19110 ts_tablet_manager.cc:1428] T a83dd71fe3404a02b7aea124048f3c90 P d4f02cff8e1a4d8f98e81b5b2e80becc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 22:33:44.805763 19116 tablet_replica.cc:439] T a83dd71fe3404a02b7aea124048f3c90 P d4f02cff8e1a4d8f98e81b5b2e80becc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1914598e5c74226b1200a6f2c694c8f" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 35253 } } peers { permanent_uuid: "d4f02cff8e1a4d8f98e81b5b2e80becc" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39683 } } peers { permanent_uuid: "83c6dfaf96b34764967f6f5497309bf1" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 43165 } } }
I20260810 22:33:44.805893 19109 tablet_bootstrap.cc:654] T a83dd71fe3404a02b7aea124048f3c90 P 83c6dfaf96b34764967f6f5497309bf1: Neither blocks nor log segments found. Creating new log.
I20260810 22:33:44.805974 19116 tablet_replica.cc:442] T a83dd71fe3404a02b7aea124048f3c90 P d4f02cff8e1a4d8f98e81b5b2e80becc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 22:33:44.809005 19109 tablet_bootstrap.cc:492] T a83dd71fe3404a02b7aea124048f3c90 P 83c6dfaf96b34764967f6f5497309bf1: No bootstrap required, opened a new log
I20260810 22:33:44.809129 19109 ts_tablet_manager.cc:1397] T a83dd71fe3404a02b7aea124048f3c90 P 83c6dfaf96b34764967f6f5497309bf1: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20260810 22:33:44.809453 19108 tablet_bootstrap.cc:492] T a83dd71fe3404a02b7aea124048f3c90 P c1914598e5c74226b1200a6f2c694c8f: No bootstrap required, opened a new log
I20260810 22:33:44.809541 19108 ts_tablet_manager.cc:1397] T a83dd71fe3404a02b7aea124048f3c90 P c1914598e5c74226b1200a6f2c694c8f: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.003s
I20260810 22:33:44.809849 19109 raft_consensus.cc:348] T a83dd71fe3404a02b7aea124048f3c90 P 83c6dfaf96b34764967f6f5497309bf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1914598e5c74226b1200a6f2c694c8f" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 35253 } } peers { permanent_uuid: "d4f02cff8e1a4d8f98e81b5b2e80becc" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39683 } } peers { permanent_uuid: "83c6dfaf96b34764967f6f5497309bf1" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 43165 } }
I20260810 22:33:44.810009 19109 raft_consensus.cc:374] T a83dd71fe3404a02b7aea124048f3c90 P 83c6dfaf96b34764967f6f5497309bf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:33:44.810068 19109 raft_consensus.cc:729] T a83dd71fe3404a02b7aea124048f3c90 P 83c6dfaf96b34764967f6f5497309bf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83c6dfaf96b34764967f6f5497309bf1, State: Initialized, Role: FOLLOWER
I20260810 22:33:44.810228 19109 consensus_queue.cc:260] T a83dd71fe3404a02b7aea124048f3c90 P 83c6dfaf96b34764967f6f5497309bf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1914598e5c74226b1200a6f2c694c8f" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 35253 } } peers { permanent_uuid: "d4f02cff8e1a4d8f98e81b5b2e80becc" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39683 } } peers { permanent_uuid: "83c6dfaf96b34764967f6f5497309bf1" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 43165 } }
I20260810 22:33:44.810482 19108 raft_consensus.cc:348] T a83dd71fe3404a02b7aea124048f3c90 P c1914598e5c74226b1200a6f2c694c8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1914598e5c74226b1200a6f2c694c8f" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 35253 } } peers { permanent_uuid: "d4f02cff8e1a4d8f98e81b5b2e80becc" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39683 } } peers { permanent_uuid: "83c6dfaf96b34764967f6f5497309bf1" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 43165 } }
I20260810 22:33:44.810571 19109 ts_tablet_manager.cc:1428] T a83dd71fe3404a02b7aea124048f3c90 P 83c6dfaf96b34764967f6f5497309bf1: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 22:33:44.810626 19108 raft_consensus.cc:374] T a83dd71fe3404a02b7aea124048f3c90 P c1914598e5c74226b1200a6f2c694c8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:33:44.810720 19108 raft_consensus.cc:729] T a83dd71fe3404a02b7aea124048f3c90 P c1914598e5c74226b1200a6f2c694c8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1914598e5c74226b1200a6f2c694c8f, State: Initialized, Role: FOLLOWER
I20260810 22:33:44.810637 19114 tablet_replica.cc:439] T a83dd71fe3404a02b7aea124048f3c90 P 83c6dfaf96b34764967f6f5497309bf1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1914598e5c74226b1200a6f2c694c8f" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 35253 } } peers { permanent_uuid: "d4f02cff8e1a4d8f98e81b5b2e80becc" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39683 } } peers { permanent_uuid: "83c6dfaf96b34764967f6f5497309bf1" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 43165 } } }
I20260810 22:33:44.810915 19114 tablet_replica.cc:442] T a83dd71fe3404a02b7aea124048f3c90 P 83c6dfaf96b34764967f6f5497309bf1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 22:33:44.810870 19108 consensus_queue.cc:260] T a83dd71fe3404a02b7aea124048f3c90 P c1914598e5c74226b1200a6f2c694c8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1914598e5c74226b1200a6f2c694c8f" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 35253 } } peers { permanent_uuid: "d4f02cff8e1a4d8f98e81b5b2e80becc" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39683 } } peers { permanent_uuid: "83c6dfaf96b34764967f6f5497309bf1" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 43165 } }
I20260810 22:33:44.811244 19108 ts_tablet_manager.cc:1428] T a83dd71fe3404a02b7aea124048f3c90 P c1914598e5c74226b1200a6f2c694c8f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260810 22:33:44.811192 19115 tablet_replica.cc:439] T a83dd71fe3404a02b7aea124048f3c90 P c1914598e5c74226b1200a6f2c694c8f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1914598e5c74226b1200a6f2c694c8f" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 35253 } } peers { permanent_uuid: "d4f02cff8e1a4d8f98e81b5b2e80becc" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39683 } } peers { permanent_uuid: "83c6dfaf96b34764967f6f5497309bf1" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 43165 } } }
I20260810 22:33:44.811373 19115 tablet_replica.cc:442] T a83dd71fe3404a02b7aea124048f3c90 P c1914598e5c74226b1200a6f2c694c8f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 22:33:44.818866 19116 raft_consensus.cc:482] T a83dd71fe3404a02b7aea124048f3c90 P d4f02cff8e1a4d8f98e81b5b2e80becc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:33:44.819026 19116 raft_consensus.cc:504] T a83dd71fe3404a02b7aea124048f3c90 P d4f02cff8e1a4d8f98e81b5b2e80becc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1914598e5c74226b1200a6f2c694c8f" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 35253 } } peers { permanent_uuid: "d4f02cff8e1a4d8f98e81b5b2e80becc" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39683 } } peers { permanent_uuid: "83c6dfaf96b34764967f6f5497309bf1" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 43165 } }
I20260810 22:33:44.820660 19116 leader_election.cc:288] T a83dd71fe3404a02b7aea124048f3c90 P d4f02cff8e1a4d8f98e81b5b2e80becc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1914598e5c74226b1200a6f2c694c8f (127.16.233.65:35253), 83c6dfaf96b34764967f6f5497309bf1 (127.16.233.67:43165)
I20260810 22:33:44.827474 19107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:44.821451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:51955 (local address 127.16.233.65:35253)
0810 22:33:44.821592 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:44.821597 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:44.822143 (+ 546us) server_negotiation.cc:407] Connection header received
0810 22:33:44.822216 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:44.822219 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:44.822288 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:44.822379 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:33:44.822800 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:44.823583 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:44.825811 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:44.826640 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:44.826780 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:44.826840 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:44.827183 (+ 343us) server_negotiation.cc:299] Negotiation successful
0810 22:33:44.827314 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 22:33:44.827472 19152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:44.820907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.65:35253 (local address 127.16.233.66:51955)
0810 22:33:44.821966 (+ 1059us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:44.821983 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:44.822131 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:44.822510 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:44.822514 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:44.822533 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:33:44.822656 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:44.822664 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:44.823771 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:44.823775 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:44.825636 (+ 1861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:44.825650 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:44.826802 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:44.826806 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:44.827004 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:44.827034 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 22:33:44.827135 (+ 101us) client_negotiation.cc:240] Negotiation successful
0810 22:33:44.827222 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":974,"thread_start_us":197,"threads_started":1}
I20260810 22:33:44.829013 18724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83dd71fe3404a02b7aea124048f3c90" candidate_uuid: "d4f02cff8e1a4d8f98e81b5b2e80becc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1914598e5c74226b1200a6f2c694c8f" is_pre_election: true
I20260810 22:33:44.829288 18724 raft_consensus.cc:2454] T a83dd71fe3404a02b7aea124048f3c90 P c1914598e5c74226b1200a6f2c694c8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4f02cff8e1a4d8f98e81b5b2e80becc in term 0.
I20260810 22:33:44.830762 18803 leader_election.cc:302] T a83dd71fe3404a02b7aea124048f3c90 P d4f02cff8e1a4d8f98e81b5b2e80becc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1914598e5c74226b1200a6f2c694c8f, d4f02cff8e1a4d8f98e81b5b2e80becc; no voters:
I20260810 22:33:44.831067 19116 raft_consensus.cc:2781] T a83dd71fe3404a02b7aea124048f3c90 P d4f02cff8e1a4d8f98e81b5b2e80becc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:33:44.831115 19116 raft_consensus.cc:482] T a83dd71fe3404a02b7aea124048f3c90 P d4f02cff8e1a4d8f98e81b5b2e80becc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:33:44.831151 19116 raft_consensus.cc:3037] T a83dd71fe3404a02b7aea124048f3c90 P d4f02cff8e1a4d8f98e81b5b2e80becc [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:33:44.833077 19106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:44.821511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:57195 (local address 127.16.233.67:43165)
0810 22:33:44.821639 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:44.821644 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:44.827744 (+ 6100us) server_negotiation.cc:407] Connection header received
0810 22:33:44.827892 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:44.827895 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:44.827950 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:44.828035 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:33:44.828605 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:44.829391 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:44.831532 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:44.832177 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:44.832277 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:44.832333 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:44.832865 (+ 532us) server_negotiation.cc:299] Negotiation successful
0810 22:33:44.832950 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 22:33:44.833333 19116 raft_consensus.cc:504] T a83dd71fe3404a02b7aea124048f3c90 P d4f02cff8e1a4d8f98e81b5b2e80becc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1914598e5c74226b1200a6f2c694c8f" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 35253 } } peers { permanent_uuid: "d4f02cff8e1a4d8f98e81b5b2e80becc" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39683 } } peers { permanent_uuid: "83c6dfaf96b34764967f6f5497309bf1" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 43165 } }
I20260810 22:33:44.833647 19151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:44.821036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.67:43165 (local address 127.16.233.66:57195)
0810 22:33:44.827544 (+ 6508us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:44.827564 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:44.827752 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:44.828194 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:44.828198 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:44.828216 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:33:44.828455 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:44.828465 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:44.829707 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:44.829710 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:44.831337 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:44.831349 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:44.832491 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:44.832495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:44.832660 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:44.832698 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 22:33:44.833280 (+ 582us) client_negotiation.cc:240] Negotiation successful
0810 22:33:44.833362 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6400,"thread_start_us":110,"threads_started":1}
I20260810 22:33:44.833717 19116 leader_election.cc:288] T a83dd71fe3404a02b7aea124048f3c90 P d4f02cff8e1a4d8f98e81b5b2e80becc [CANDIDATE]: Term 1 election: Requested vote from peers c1914598e5c74226b1200a6f2c694c8f (127.16.233.65:35253), 83c6dfaf96b34764967f6f5497309bf1 (127.16.233.67:43165)
I20260810 22:33:44.834384 19032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83dd71fe3404a02b7aea124048f3c90" candidate_uuid: "d4f02cff8e1a4d8f98e81b5b2e80becc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c6dfaf96b34764967f6f5497309bf1" is_pre_election: true
I20260810 22:33:44.834548 19032 raft_consensus.cc:2454] T a83dd71fe3404a02b7aea124048f3c90 P 83c6dfaf96b34764967f6f5497309bf1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4f02cff8e1a4d8f98e81b5b2e80becc in term 0.
I20260810 22:33:44.834628 18724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83dd71fe3404a02b7aea124048f3c90" candidate_uuid: "d4f02cff8e1a4d8f98e81b5b2e80becc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1914598e5c74226b1200a6f2c694c8f"
I20260810 22:33:44.834748 18724 raft_consensus.cc:3037] T a83dd71fe3404a02b7aea124048f3c90 P c1914598e5c74226b1200a6f2c694c8f [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:33:44.834869 19031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83dd71fe3404a02b7aea124048f3c90" candidate_uuid: "d4f02cff8e1a4d8f98e81b5b2e80becc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c6dfaf96b34764967f6f5497309bf1"
I20260810 22:33:44.834981 19031 raft_consensus.cc:3037] T a83dd71fe3404a02b7aea124048f3c90 P 83c6dfaf96b34764967f6f5497309bf1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:33:44.836167 18724 raft_consensus.cc:2454] T a83dd71fe3404a02b7aea124048f3c90 P c1914598e5c74226b1200a6f2c694c8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4f02cff8e1a4d8f98e81b5b2e80becc in term 1.
I20260810 22:33:44.836743 19031 raft_consensus.cc:2454] T a83dd71fe3404a02b7aea124048f3c90 P 83c6dfaf96b34764967f6f5497309bf1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4f02cff8e1a4d8f98e81b5b2e80becc in term 1.
I20260810 22:33:44.837641 18803 leader_election.cc:302] T a83dd71fe3404a02b7aea124048f3c90 P d4f02cff8e1a4d8f98e81b5b2e80becc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1914598e5c74226b1200a6f2c694c8f, d4f02cff8e1a4d8f98e81b5b2e80becc; no voters:
I20260810 22:33:44.838135 19116 raft_consensus.cc:2781] T a83dd71fe3404a02b7aea124048f3c90 P d4f02cff8e1a4d8f98e81b5b2e80becc [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:33:44.838383 19116 raft_consensus.cc:686] T a83dd71fe3404a02b7aea124048f3c90 P d4f02cff8e1a4d8f98e81b5b2e80becc [term 1 LEADER]: Becoming Leader. State: Replica: d4f02cff8e1a4d8f98e81b5b2e80becc, State: Running, Role: LEADER
I20260810 22:33:44.838632 19116 consensus_queue.cc:237] T a83dd71fe3404a02b7aea124048f3c90 P d4f02cff8e1a4d8f98e81b5b2e80becc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1914598e5c74226b1200a6f2c694c8f" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 35253 } } peers { permanent_uuid: "d4f02cff8e1a4d8f98e81b5b2e80becc" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39683 } } peers { permanent_uuid: "83c6dfaf96b34764967f6f5497309bf1" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 43165 } }
I20260810 22:33:44.840348 19153 tablet_replica.cc:439] T a83dd71fe3404a02b7aea124048f3c90 P d4f02cff8e1a4d8f98e81b5b2e80becc: TxnStatusTablet state changed. Reason: New leader d4f02cff8e1a4d8f98e81b5b2e80becc. Latest consensus state: current_term: 1 leader_uuid: "d4f02cff8e1a4d8f98e81b5b2e80becc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1914598e5c74226b1200a6f2c694c8f" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 35253 } } peers { permanent_uuid: "d4f02cff8e1a4d8f98e81b5b2e80becc" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39683 } } peers { permanent_uuid: "83c6dfaf96b34764967f6f5497309bf1" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 43165 } } }
I20260810 22:33:44.840464 19153 tablet_replica.cc:442] T a83dd71fe3404a02b7aea124048f3c90 P d4f02cff8e1a4d8f98e81b5b2e80becc: This TxnStatusTablet replica's current role is: LEADER
I20260810 22:33:44.841286 19156 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 22:33:44.841524 19156 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 22:33:44.841570 18552 catalog_manager.cc:5495] T a83dd71fe3404a02b7aea124048f3c90 P d4f02cff8e1a4d8f98e81b5b2e80becc reported cstate change: term changed from 0 to 1, leader changed from <none> to d4f02cff8e1a4d8f98e81b5b2e80becc (127.16.233.66). New cstate: current_term: 1 leader_uuid: "d4f02cff8e1a4d8f98e81b5b2e80becc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1914598e5c74226b1200a6f2c694c8f" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 35253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4f02cff8e1a4d8f98e81b5b2e80becc" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39683 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83c6dfaf96b34764967f6f5497309bf1" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 43165 } health_report { overall_health: UNKNOWN } } }
I20260810 22:33:44.858186 19103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:44.852383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40086 (local address 127.16.233.66:39683)
0810 22:33:44.852533 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:44.852538 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:44.852555 (+ 17us) server_negotiation.cc:407] Connection header received
0810 22:33:44.852746 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:44.852750 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:44.852883 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:44.853029 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:33:44.853678 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:44.854624 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:44.855938 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:44.856365 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:44.856499 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:44.856895 (+ 396us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:33:44.857683 (+ 788us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:33:44.857813 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:44.857948 (+ 135us) server_negotiation.cc:299] Negotiation successful
0810 22:33:44.858048 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 22:33:44.858181 19146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:44.852251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.66:39683 (local address 127.0.0.1:40086)
0810 22:33:44.852436 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:44.852456 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:44.852570 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:44.853066 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:44.853070 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:44.853079 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 22:33:44.853207 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:44.853214 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:44.854770 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:44.854774 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:44.855773 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:44.855785 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:44.856516 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:44.856520 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:44.856615 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:44.856673 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 22:33:44.857804 (+ 1131us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 22:33:44.857831 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 22:33:44.857907 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 22:33:44.858013 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260810 22:33:44.862082 19031 raft_consensus.cc:1261] T a83dd71fe3404a02b7aea124048f3c90 P 83c6dfaf96b34764967f6f5497309bf1 [term 1 FOLLOWER]: Refusing update from remote peer d4f02cff8e1a4d8f98e81b5b2e80becc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:33:44.862417 18724 raft_consensus.cc:1261] T a83dd71fe3404a02b7aea124048f3c90 P c1914598e5c74226b1200a6f2c694c8f [term 1 FOLLOWER]: Refusing update from remote peer d4f02cff8e1a4d8f98e81b5b2e80becc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:33:44.862952 19153 consensus_queue.cc:1035] T a83dd71fe3404a02b7aea124048f3c90 P d4f02cff8e1a4d8f98e81b5b2e80becc [LEADER]: Connected to new peer: Peer: permanent_uuid: "83c6dfaf96b34764967f6f5497309bf1" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 43165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:33:44.863232 19116 consensus_queue.cc:1035] T a83dd71fe3404a02b7aea124048f3c90 P d4f02cff8e1a4d8f98e81b5b2e80becc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1914598e5c74226b1200a6f2c694c8f" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 35253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:33:44.866560 19115 tablet_replica.cc:439] T a83dd71fe3404a02b7aea124048f3c90 P c1914598e5c74226b1200a6f2c694c8f: TxnStatusTablet state changed. Reason: New leader d4f02cff8e1a4d8f98e81b5b2e80becc. Latest consensus state: current_term: 1 leader_uuid: "d4f02cff8e1a4d8f98e81b5b2e80becc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1914598e5c74226b1200a6f2c694c8f" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 35253 } } peers { permanent_uuid: "d4f02cff8e1a4d8f98e81b5b2e80becc" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39683 } } peers { permanent_uuid: "83c6dfaf96b34764967f6f5497309bf1" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 43165 } } }
I20260810 22:33:44.866842 19115 tablet_replica.cc:442] T a83dd71fe3404a02b7aea124048f3c90 P c1914598e5c74226b1200a6f2c694c8f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 22:33:44.867605 19114 tablet_replica.cc:439] T a83dd71fe3404a02b7aea124048f3c90 P 83c6dfaf96b34764967f6f5497309bf1: TxnStatusTablet state changed. Reason: New leader d4f02cff8e1a4d8f98e81b5b2e80becc. Latest consensus state: current_term: 1 leader_uuid: "d4f02cff8e1a4d8f98e81b5b2e80becc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1914598e5c74226b1200a6f2c694c8f" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 35253 } } peers { permanent_uuid: "d4f02cff8e1a4d8f98e81b5b2e80becc" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39683 } } peers { permanent_uuid: "83c6dfaf96b34764967f6f5497309bf1" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 43165 } } }
I20260810 22:33:44.867800 19114 tablet_replica.cc:442] T a83dd71fe3404a02b7aea124048f3c90 P 83c6dfaf96b34764967f6f5497309bf1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 22:33:44.870712 19116 tablet_replica.cc:439] T a83dd71fe3404a02b7aea124048f3c90 P d4f02cff8e1a4d8f98e81b5b2e80becc: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d4f02cff8e1a4d8f98e81b5b2e80becc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1914598e5c74226b1200a6f2c694c8f" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 35253 } } peers { permanent_uuid: "d4f02cff8e1a4d8f98e81b5b2e80becc" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39683 } } peers { permanent_uuid: "83c6dfaf96b34764967f6f5497309bf1" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 43165 } } }
I20260810 22:33:44.870875 19116 tablet_replica.cc:442] T a83dd71fe3404a02b7aea124048f3c90 P d4f02cff8e1a4d8f98e81b5b2e80becc: This TxnStatusTablet replica's current role is: LEADER
I20260810 22:33:44.871304 19114 tablet_replica.cc:439] T a83dd71fe3404a02b7aea124048f3c90 P 83c6dfaf96b34764967f6f5497309bf1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d4f02cff8e1a4d8f98e81b5b2e80becc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1914598e5c74226b1200a6f2c694c8f" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 35253 } } peers { permanent_uuid: "d4f02cff8e1a4d8f98e81b5b2e80becc" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39683 } } peers { permanent_uuid: "83c6dfaf96b34764967f6f5497309bf1" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 43165 } } }
I20260810 22:33:44.871404 19114 tablet_replica.cc:442] T a83dd71fe3404a02b7aea124048f3c90 P 83c6dfaf96b34764967f6f5497309bf1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 22:33:44.876382 19116 tablet_replica.cc:439] T a83dd71fe3404a02b7aea124048f3c90 P d4f02cff8e1a4d8f98e81b5b2e80becc: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d4f02cff8e1a4d8f98e81b5b2e80becc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1914598e5c74226b1200a6f2c694c8f" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 35253 } } peers { permanent_uuid: "d4f02cff8e1a4d8f98e81b5b2e80becc" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39683 } } peers { permanent_uuid: "83c6dfaf96b34764967f6f5497309bf1" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 43165 } } }
I20260810 22:33:44.876523 19116 tablet_replica.cc:442] T a83dd71fe3404a02b7aea124048f3c90 P d4f02cff8e1a4d8f98e81b5b2e80becc: This TxnStatusTablet replica's current role is: LEADER
I20260810 22:33:44.878288 19115 tablet_replica.cc:439] T a83dd71fe3404a02b7aea124048f3c90 P c1914598e5c74226b1200a6f2c694c8f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d4f02cff8e1a4d8f98e81b5b2e80becc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1914598e5c74226b1200a6f2c694c8f" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 35253 } } peers { permanent_uuid: "d4f02cff8e1a4d8f98e81b5b2e80becc" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39683 } } peers { permanent_uuid: "83c6dfaf96b34764967f6f5497309bf1" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 43165 } } }
I20260810 22:33:44.878451 19115 tablet_replica.cc:442] T a83dd71fe3404a02b7aea124048f3c90 P c1914598e5c74226b1200a6f2c694c8f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 22:33:44.879150 19122 mvcc.cc:205] Tried to move back new op lower bound from 7317099417029201920 to 7317099416942026752. Current Snapshot: MvccSnapshot[applied={T|T < 7317099417029201920 or (T in {7317099417029201920})}]
I20260810 22:33:44.890903 19103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:44.886808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:41225 (local address 127.16.233.66:39683)
0810 22:33:44.886958 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:44.886964 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:44.887498 (+ 534us) server_negotiation.cc:407] Connection header received
0810 22:33:44.887677 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:44.887681 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:44.887740 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:44.887830 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:33:44.888139 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:44.888667 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:44.889703 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:44.889941 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:44.890067 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:44.890390 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:33:44.890478 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:33:44.890567 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:44.890681 (+ 114us) server_negotiation.cc:299] Negotiation successful
0810 22:33:44.890740 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 22:33:44.890914 19164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:44.886700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.66:39683 (local address 127.16.233.65:41225)
0810 22:33:44.887378 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:44.887393 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:44.887549 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:44.887858 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:44.887862 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:44.887871 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 22:33:44.888006 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:44.888013 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:44.888799 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:44.888806 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:44.889550 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:44.889561 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:44.890071 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:44.890075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:44.890173 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:44.890236 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 22:33:44.890585 (+ 349us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 22:33:44.890594 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 22:33:44.890672 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 22:33:44.890772 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":333,"threads_started":1}
I20260810 22:33:44.899613 18877 raft_consensus.cc:1261] T b22fdf1a39cf4f098b3a987e9bae6d83 P d4f02cff8e1a4d8f98e81b5b2e80becc [term 1 FOLLOWER]: Refusing update from remote peer c1914598e5c74226b1200a6f2c694c8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:33:44.900267 19031 raft_consensus.cc:1261] T b22fdf1a39cf4f098b3a987e9bae6d83 P 83c6dfaf96b34764967f6f5497309bf1 [term 1 FOLLOWER]: Refusing update from remote peer c1914598e5c74226b1200a6f2c694c8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:33:44.900398 19119 consensus_queue.cc:1035] T b22fdf1a39cf4f098b3a987e9bae6d83 P c1914598e5c74226b1200a6f2c694c8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4f02cff8e1a4d8f98e81b5b2e80becc" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:33:44.901003 19115 consensus_queue.cc:1035] T b22fdf1a39cf4f098b3a987e9bae6d83 P c1914598e5c74226b1200a6f2c694c8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "83c6dfaf96b34764967f6f5497309bf1" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 43165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:33:44.934516 19107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:44.928228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:58595 (local address 127.16.233.65:35253)
0810 22:33:44.928401 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:44.928407 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:44.928756 (+ 349us) server_negotiation.cc:407] Connection header received
0810 22:33:44.928987 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:44.928991 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:44.929074 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:44.929193 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:33:44.929673 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:44.930490 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:44.932494 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:44.932977 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:44.933101 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:44.933794 (+ 693us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:33:44.933966 (+ 172us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:33:44.934022 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:44.934244 (+ 222us) server_negotiation.cc:299] Negotiation successful
0810 22:33:44.934340 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 22:33:44.934656 19169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:44.928094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.65:35253 (local address 127.16.233.66:58595)
0810 22:33:44.928624 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:44.928641 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:44.928767 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:44.929207 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:44.929210 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:44.929220 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 22:33:44.929525 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:44.929533 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:44.930670 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:44.930675 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:44.932281 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:44.932294 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:44.933198 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:44.933202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:44.933454 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:44.933545 (+ 91us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 22:33:44.934099 (+ 554us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 22:33:44.934109 (+ 10us) client_negotiation.cc:769] Sending connection context
0810 22:33:44.934207 (+ 98us) client_negotiation.cc:240] Negotiation successful
0810 22:33:44.934317 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":217,"threads_started":1}
I20260810 22:33:44.957233 18552 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:33:44.957410 18552 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:33:44.959532 18552 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 04121952a88449349c980197e530ae20: Sending DeleteTablet for 3 replicas of tablet b22fdf1a39cf4f098b3a987e9bae6d83
I20260810 22:33:44.960283 19012 tablet_service.cc:1514] Processing DeleteTablet for tablet b22fdf1a39cf4f098b3a987e9bae6d83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:33:44 UTC) from {username='kudu', principal='kudu/127.16.233.126@KRBTEST.COM'} at 127.0.0.1:36986
I20260810 22:33:44.960486 19134 tablet_replica.cc:330] T b22fdf1a39cf4f098b3a987e9bae6d83 P 83c6dfaf96b34764967f6f5497309bf1: stopping tablet replica
I20260810 22:33:44.960630 19134 raft_consensus.cc:2229] T b22fdf1a39cf4f098b3a987e9bae6d83 P 83c6dfaf96b34764967f6f5497309bf1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:33:44.960768 19134 raft_consensus.cc:2258] T b22fdf1a39cf4f098b3a987e9bae6d83 P 83c6dfaf96b34764967f6f5497309bf1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:33:44.960995 18703 tablet_service.cc:1514] Processing DeleteTablet for tablet b22fdf1a39cf4f098b3a987e9bae6d83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:33:44 UTC) from {username='kudu', principal='kudu/127.16.233.126@KRBTEST.COM'} at 127.0.0.1:50092
I20260810 22:33:44.961203 19133 tablet_replica.cc:330] T b22fdf1a39cf4f098b3a987e9bae6d83 P c1914598e5c74226b1200a6f2c694c8f: stopping tablet replica
I20260810 22:33:44.961364 19133 raft_consensus.cc:2229] T b22fdf1a39cf4f098b3a987e9bae6d83 P c1914598e5c74226b1200a6f2c694c8f [term 1 LEADER]: Raft consensus shutting down.
I20260810 22:33:44.961563 19133 raft_consensus.cc:2258] T b22fdf1a39cf4f098b3a987e9bae6d83 P c1914598e5c74226b1200a6f2c694c8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:33:44.961611 18855 tablet_service.cc:1514] Processing DeleteTablet for tablet b22fdf1a39cf4f098b3a987e9bae6d83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:33:44 UTC) from {username='kudu', principal='kudu/127.16.233.126@KRBTEST.COM'} at 127.0.0.1:40080
I20260810 22:33:44.961719 19134 ts_tablet_manager.cc:1905] T b22fdf1a39cf4f098b3a987e9bae6d83 P 83c6dfaf96b34764967f6f5497309bf1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:33:44.961840 19132 tablet_replica.cc:330] T b22fdf1a39cf4f098b3a987e9bae6d83 P d4f02cff8e1a4d8f98e81b5b2e80becc: stopping tablet replica
I20260810 22:33:44.961925 19132 raft_consensus.cc:2229] T b22fdf1a39cf4f098b3a987e9bae6d83 P d4f02cff8e1a4d8f98e81b5b2e80becc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:33:44.962014 19132 raft_consensus.cc:2258] T b22fdf1a39cf4f098b3a987e9bae6d83 P d4f02cff8e1a4d8f98e81b5b2e80becc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:33:44.962796 19132 ts_tablet_manager.cc:1905] T b22fdf1a39cf4f098b3a987e9bae6d83 P d4f02cff8e1a4d8f98e81b5b2e80becc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:33:44.962786 19133 ts_tablet_manager.cc:1905] T b22fdf1a39cf4f098b3a987e9bae6d83 P c1914598e5c74226b1200a6f2c694c8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:33:44.964916 19134 ts_tablet_manager.cc:1918] T b22fdf1a39cf4f098b3a987e9bae6d83 P 83c6dfaf96b34764967f6f5497309bf1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 22:33:44.964998 19134 log.cc:1196] T b22fdf1a39cf4f098b3a987e9bae6d83 P 83c6dfaf96b34764967f6f5497309bf1: Deleting WAL directory at /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/ts-2/wal/wals/b22fdf1a39cf4f098b3a987e9bae6d83
I20260810 22:33:44.965301 19134 ts_tablet_manager.cc:1939] T b22fdf1a39cf4f098b3a987e9bae6d83 P 83c6dfaf96b34764967f6f5497309bf1: Deleting consensus metadata
I20260810 22:33:44.965637 19132 ts_tablet_manager.cc:1918] T b22fdf1a39cf4f098b3a987e9bae6d83 P d4f02cff8e1a4d8f98e81b5b2e80becc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 22:33:44.965718 19132 log.cc:1196] T b22fdf1a39cf4f098b3a987e9bae6d83 P d4f02cff8e1a4d8f98e81b5b2e80becc: Deleting WAL directory at /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/ts-1/wal/wals/b22fdf1a39cf4f098b3a987e9bae6d83
I20260810 22:33:44.965871 19133 ts_tablet_manager.cc:1918] T b22fdf1a39cf4f098b3a987e9bae6d83 P c1914598e5c74226b1200a6f2c694c8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 22:33:44.965945 19133 log.cc:1196] T b22fdf1a39cf4f098b3a987e9bae6d83 P c1914598e5c74226b1200a6f2c694c8f: Deleting WAL directory at /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786401215402979-17317-0/minicluster-data/ts-0/wal/wals/b22fdf1a39cf4f098b3a987e9bae6d83
I20260810 22:33:44.966006 19132 ts_tablet_manager.cc:1939] T b22fdf1a39cf4f098b3a987e9bae6d83 P d4f02cff8e1a4d8f98e81b5b2e80becc: Deleting consensus metadata
I20260810 22:33:44.966125 18536 catalog_manager.cc:4841] TS 83c6dfaf96b34764967f6f5497309bf1 (127.16.233.67:43165): tablet b22fdf1a39cf4f098b3a987e9bae6d83 (table test-table [id=df1c088b750544a3af78a988aa7dc433]) successfully deleted
I20260810 22:33:44.966203 19133 ts_tablet_manager.cc:1939] T b22fdf1a39cf4f098b3a987e9bae6d83 P c1914598e5c74226b1200a6f2c694c8f: Deleting consensus metadata
I20260810 22:33:44.967082 18536 catalog_manager.cc:4841] TS d4f02cff8e1a4d8f98e81b5b2e80becc (127.16.233.66:39683): tablet b22fdf1a39cf4f098b3a987e9bae6d83 (table test-table [id=df1c088b750544a3af78a988aa7dc433]) successfully deleted
I20260810 22:33:44.967453 18537 catalog_manager.cc:4841] TS c1914598e5c74226b1200a6f2c694c8f (127.16.233.65:35253): tablet b22fdf1a39cf4f098b3a987e9bae6d83 (table test-table [id=df1c088b750544a3af78a988aa7dc433]) successfully deleted
Aug 10 22:33:44 dist-test-slave-77v9 krb5kdc[18499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401224, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.233.65@KRBTEST.COM
I20260810 22:33:44.978799 19107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:44.966833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50106 (local address 127.16.233.65:35253)
0810 22:33:44.966960 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:44.966965 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:44.967275 (+ 310us) server_negotiation.cc:407] Connection header received
0810 22:33:44.967386 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:44.967389 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:44.967446 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:44.967537 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:44.968301 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:44.969203 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:44.970145 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:44.970536 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:44.970651 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:44.974341 (+ 3690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:44.974357 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:44.974378 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:44.974410 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:44.976656 (+ 2246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:44.977256 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:44.977264 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:44.977267 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:44.977378 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:44.977682 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:44.977685 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:44.977687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:44.978047 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:44.978292 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:44.978458 (+ 166us) server_negotiation.cc:299] Negotiation successful
0810 22:33:44.978612 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
W20260810 22:33:44.979451 18768 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50106
I20260810 22:33:44.990157 18606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:44.981755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40390 (local address 127.16.233.126:43649)
0810 22:33:44.981908 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:44.981913 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:44.981928 (+ 15us) server_negotiation.cc:407] Connection header received
0810 22:33:44.982066 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:44.982069 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:44.982121 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:44.982208 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:44.982924 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:44.983735 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:44.984574 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:44.985007 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:44.985117 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:44.986674 (+ 1557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:44.986678 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:44.986682 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:44.986715 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:44.988380 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:44.988858 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:44.988862 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:44.988864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:44.988959 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:44.989249 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:44.989253 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:44.989255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:44.989485 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:44.989616 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:44.989857 (+ 241us) server_negotiation.cc:299] Negotiation successful
0810 22:33:44.989985 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260810 22:33:44.990679 18552 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40390
I20260810 22:33:44.992174 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 18632
I20260810 22:33:45.005515 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 18784
I20260810 22:33:45.014627 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 18937
I20260810 22:33:45.024962 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 18516
2026-08-10T22:33:45Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3058 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:33:45 dist-test-slave-77v9 krb5kdc[19178](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:33:45 dist-test-slave-77v9 krb5kdc[19178](info): set up 2 sockets
Aug 10 22:33:45 dist-test-slave-77v9 krb5kdc[19178](info): commencing operation
krb5kdc: starting...
W20260810 22:33:46.168699 17317 mini_kdc.cc:121] Time spent starting KDC: real 1.117s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Aug 10 22:33:46 dist-test-slave-77v9 krb5kdc[19178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:33:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:33:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.233.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab.
Entry for principal kudu/127.16.233.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab.
I20260810 22:33:46.308396 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:43475
--webserver_interface=127.16.233.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab
--principal=kudu/127.16.233.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:34979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.126:43475
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:33:46.428591 19192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:33:46.429023 19192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:33:46.429119 19192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:33:46.433271 19192 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:33:46.433383 19192 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 22:33:46.433416 19192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:33:46.433460 19192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:33:46.433511 19192 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:33:46.433545 19192 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:33:46.439406 19192 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:34979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.126:43475
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab
--principal=kudu/127.16.233.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:43475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:33:46.440658 19192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:33:46.441797 19192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:33:46.451176 19199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:33:46.451285 19198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:33:46.451510 19192 server_base.cc:1034] running on GCE node
W20260810 22:33:46.451285 19201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:33:46.452147 19192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:33:46.453545 19192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:33:46.454747 19192 hybrid_clock.cc:648] HybridClock initialized: now 1786401226454735 us; error 31 us; skew 500 ppm
Aug 10 22:33:46 dist-test-slave-77v9 krb5kdc[19178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401226, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:33:46.464138 19192 init.cc:377] Logged in from keytab as kudu/127.16.233.126@KRBTEST.COM (short username kudu)
I20260810 22:33:46.465353 19192 webserver.cc:460] Webserver started at http://127.16.233.126:34625/ using document root <none> and password file <none>
I20260810 22:33:46.465984 19192 fs_manager.cc:359] Metadata directory not provided
I20260810 22:33:46.466091 19192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:33:46.466341 19192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:33:46.468585 19192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/master-0/data/instance:
uuid: "5652ecc5dafa4979ab57ef1506d00660"
format_stamp: "Formatted at 2026-08-10 22:33:46 on dist-test-slave-77v9"
server_key: "138bf817d4079aa74edee5390f9c2727"
server_key_iv: "4fa7532cc0e55d8f54204c8ef3863f27"
server_key_version: "encryptionkey@0"
I20260810 22:33:46.469285 19192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/master-0/wal/instance:
uuid: "5652ecc5dafa4979ab57ef1506d00660"
format_stamp: "Formatted at 2026-08-10 22:33:46 on dist-test-slave-77v9"
server_key: "138bf817d4079aa74edee5390f9c2727"
server_key_iv: "4fa7532cc0e55d8f54204c8ef3863f27"
server_key_version: "encryptionkey@0"
I20260810 22:33:46.473381 19192 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 22:33:46.476214 19209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:46.477782 19192 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:33:46.477946 19192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/master-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/master-0/wal
uuid: "5652ecc5dafa4979ab57ef1506d00660"
format_stamp: "Formatted at 2026-08-10 22:33:46 on dist-test-slave-77v9"
server_key: "138bf817d4079aa74edee5390f9c2727"
server_key_iv: "4fa7532cc0e55d8f54204c8ef3863f27"
server_key_version: "encryptionkey@0"
I20260810 22:33:46.478081 19192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:33:46.507305 19192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:33:46.511118 19192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:33:46.511363 19192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:33:46.521842 19192 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.126:43475
I20260810 22:33:46.521858 19277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.126:43475 every 8 connection(s)
I20260810 22:33:46.523020 19192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
I20260810 22:33:46.526176 19278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:33:46.527168 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 19192
I20260810 22:33:46.527292 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/master-0/wal/instance
I20260810 22:33:46.527590 17317 external_mini_cluster.cc:1428] Setting key 39a1d23dfe2db08d64f4cf1325b60d0d
I20260810 22:33:46.532706 19278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5652ecc5dafa4979ab57ef1506d00660: Bootstrap starting.
I20260810 22:33:46.535138 19278 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5652ecc5dafa4979ab57ef1506d00660: Neither blocks nor log segments found. Creating new log.
I20260810 22:33:46.535921 19278 log.cc:824] T 00000000000000000000000000000000 P 5652ecc5dafa4979ab57ef1506d00660: Log is configured to *not* fsync() on all Append() calls
Aug 10 22:33:46 dist-test-slave-77v9 krb5kdc[19178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:33:46.538281 19278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5652ecc5dafa4979ab57ef1506d00660: No bootstrap required, opened a new log
I20260810 22:33:46.541918 19278 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5652ecc5dafa4979ab57ef1506d00660 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5652ecc5dafa4979ab57ef1506d00660" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 43475 } }
I20260810 22:33:46.542251 19278 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5652ecc5dafa4979ab57ef1506d00660 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:33:46.542349 19278 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5652ecc5dafa4979ab57ef1506d00660 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5652ecc5dafa4979ab57ef1506d00660, State: Initialized, Role: FOLLOWER
I20260810 22:33:46.542891 19278 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5652ecc5dafa4979ab57ef1506d00660 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5652ecc5dafa4979ab57ef1506d00660" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 43475 } }
I20260810 22:33:46.543057 19278 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5652ecc5dafa4979ab57ef1506d00660 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:33:46.543181 19278 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5652ecc5dafa4979ab57ef1506d00660 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:33:46.543300 19278 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5652ecc5dafa4979ab57ef1506d00660 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:33:46.544621 19278 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5652ecc5dafa4979ab57ef1506d00660 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5652ecc5dafa4979ab57ef1506d00660" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 43475 } }
I20260810 22:33:46.544768 19281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:46.529278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34290 (local address 127.16.233.126:43475)
0810 22:33:46.529800 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:46.529818 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:46.529857 (+ 39us) server_negotiation.cc:407] Connection header received
0810 22:33:46.530533 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:46.530551 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:46.530892 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:46.531267 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:46.532075 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:46.533173 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:46.534116 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:46.534610 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:46.534819 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:46.537983 (+ 3164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:46.537990 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:46.538006 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:46.538038 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:46.541010 (+ 2972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:46.541631 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:46.541638 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:46.541644 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:46.541784 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:46.542140 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:46.542144 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:46.542146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:46.542730 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:46.542980 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:46.543330 (+ 350us) server_negotiation.cc:299] Negotiation successful
0810 22:33:46.543689 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":154,"threads_started":1}
I20260810 22:33:46.545034 19278 leader_election.cc:302] T 00000000000000000000000000000000 P 5652ecc5dafa4979ab57ef1506d00660 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5652ecc5dafa4979ab57ef1506d00660; no voters:
I20260810 22:33:46.545430 19278 leader_election.cc:288] T 00000000000000000000000000000000 P 5652ecc5dafa4979ab57ef1506d00660 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:33:46.545612 19283 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5652ecc5dafa4979ab57ef1506d00660 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:33:46.545918 19283 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5652ecc5dafa4979ab57ef1506d00660 [term 1 LEADER]: Becoming Leader. State: Replica: 5652ecc5dafa4979ab57ef1506d00660, State: Running, Role: LEADER
I20260810 22:33:46.546253 19283 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5652ecc5dafa4979ab57ef1506d00660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5652ecc5dafa4979ab57ef1506d00660" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 43475 } }
I20260810 22:33:46.546787 19278 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5652ecc5dafa4979ab57ef1506d00660 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:33:46.547458 19283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5652ecc5dafa4979ab57ef1506d00660 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5652ecc5dafa4979ab57ef1506d00660. Latest consensus state: current_term: 1 leader_uuid: "5652ecc5dafa4979ab57ef1506d00660" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5652ecc5dafa4979ab57ef1506d00660" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 43475 } } }
I20260810 22:33:46.547590 19283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5652ecc5dafa4979ab57ef1506d00660 [sys.catalog]: This master's current role is: LEADER
I20260810 22:33:46.548056 19289 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:33:46.548173 19284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5652ecc5dafa4979ab57ef1506d00660 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5652ecc5dafa4979ab57ef1506d00660" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5652ecc5dafa4979ab57ef1506d00660" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 43475 } } }
I20260810 22:33:46.548321 19284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5652ecc5dafa4979ab57ef1506d00660 [sys.catalog]: This master's current role is: LEADER
I20260810 22:33:46.551420 19289 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:33:46.558133 19289 catalog_manager.cc:1347] Generated new cluster ID: 6f31eca7de6a4ec89fd84aa5ff6326fd
I20260810 22:33:46.558231 19289 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:33:46.579237 19289 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:33:46.580578 19289 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:33:46.585210 19289 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5652ecc5dafa4979ab57ef1506d00660: Generated new TSK 0
I20260810 22:33:46.585988 19289 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.233.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab.
Entry for principal kudu/127.16.233.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab.
I20260810 22:33:46.625080 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.65:0
--local_ip_for_outbound_sockets=127.16.233.65
--webserver_interface=127.16.233.65
--webserver_port=0
--tserver_master_addrs=127.16.233.126:43475
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab
--principal=kudu/127.16.233.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:34979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:33:46.746632 19306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:33:46.746991 19306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:33:46.747082 19306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:33:46.751565 19306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:33:46.751644 19306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:33:46.751693 19306 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 22:33:46.751709 19306 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 22:33:46.751812 19306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.65
I20260810 22:33:46.757604 19306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:34979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab
--principal=kudu/127.16.233.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.233.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.233.126:43475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.65
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:33:46.758746 19306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:33:46.759743 19306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:33:46.770269 19313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:33:46.770288 19312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:33:46.770337 19315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:33:46.770637 19306 server_base.cc:1034] running on GCE node
I20260810 22:33:46.771205 19306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:33:46.771915 19306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:33:46.773116 19306 hybrid_clock.cc:648] HybridClock initialized: now 1786401226773102 us; error 42 us; skew 500 ppm
Aug 10 22:33:46 dist-test-slave-77v9 krb5kdc[19178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401226, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:33:46.776255 19306 init.cc:377] Logged in from keytab as kudu/127.16.233.65@KRBTEST.COM (short username kudu)
I20260810 22:33:46.777423 19306 webserver.cc:460] Webserver started at http://127.16.233.65:42081/ using document root <none> and password file <none>
I20260810 22:33:46.778043 19306 fs_manager.cc:359] Metadata directory not provided
I20260810 22:33:46.778141 19306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:33:46.778395 19306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:33:46.780579 19306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/ts-0/data/instance:
uuid: "5d3b49686ca5430db55bc282694fb9d4"
format_stamp: "Formatted at 2026-08-10 22:33:46 on dist-test-slave-77v9"
server_key: "8367d5311dbf533763f2fde0e62c0f6b"
server_key_iv: "505c409795708d843acf829764bbece2"
server_key_version: "encryptionkey@0"
I20260810 22:33:46.781241 19306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/ts-0/wal/instance:
uuid: "5d3b49686ca5430db55bc282694fb9d4"
format_stamp: "Formatted at 2026-08-10 22:33:46 on dist-test-slave-77v9"
server_key: "8367d5311dbf533763f2fde0e62c0f6b"
server_key_iv: "505c409795708d843acf829764bbece2"
server_key_version: "encryptionkey@0"
I20260810 22:33:46.785698 19306 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 22:33:46.788471 19323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:46.790076 19306 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 22:33:46.790271 19306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/ts-0/wal
uuid: "5d3b49686ca5430db55bc282694fb9d4"
format_stamp: "Formatted at 2026-08-10 22:33:46 on dist-test-slave-77v9"
server_key: "8367d5311dbf533763f2fde0e62c0f6b"
server_key_iv: "505c409795708d843acf829764bbece2"
server_key_version: "encryptionkey@0"
I20260810 22:33:46.790429 19306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:33:46.811407 19306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:33:46.815032 19306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:33:46.815267 19306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:33:46.816829 19306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:33:46.816918 19306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:46.816992 19306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:33:46.817044 19306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 22:33:46 dist-test-slave-77v9 krb5kdc[19178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401226, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.65@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:33:46.828766 19306 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.65:43731
I20260810 22:33:46.828784 19460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.65:43731 every 8 connection(s)
I20260810 22:33:46.830080 19306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
I20260810 22:33:46.832801 19281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:46.818193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:57243 (local address 127.16.233.126:43475)
0810 22:33:46.818365 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:46.818371 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:46.819328 (+ 957us) server_negotiation.cc:407] Connection header received
0810 22:33:46.820634 (+ 1306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:46.820638 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:46.820714 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:46.820865 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:46.822380 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:46.822990 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:46.824525 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:46.824950 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:46.825081 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:46.828451 (+ 3370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:46.828455 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:46.828458 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:46.828495 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:46.830383 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:46.831221 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:46.831225 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:46.831227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:46.831329 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:46.831700 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:46.831702 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:46.831704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:46.831954 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:46.832029 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:46.832560 (+ 531us) server_negotiation.cc:299] Negotiation successful
0810 22:33:46.832677 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 22:33:46.833454 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 19306
I20260810 22:33:46.833582 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/ts-0/wal/instance
I20260810 22:33:46.833865 17317 external_mini_cluster.cc:1428] Setting key a94dff1b3795791d49d8d7cacc062541
I20260810 22:33:46.834020 19333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:46.818620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:43475 (local address 127.16.233.65:57243)
0810 22:33:46.819138 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:46.819197 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:46.820390 (+ 1193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:46.821062 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:46.821074 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:46.821644 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:33:46.822199 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:46.822215 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:46.823140 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:46.823145 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:46.824380 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:46.824391 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:46.825164 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:46.825169 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:46.825423 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:46.826108 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:33:46.826131 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:33:46.828210 (+ 2079us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:33:46.830544 (+ 2334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:46.830551 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:46.830569 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:46.831086 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:46.831443 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:46.831446 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:46.831447 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:46.831595 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:46.832078 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:33:46.832083 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:33:46.832340 (+ 257us) client_negotiation.cc:769] Sending connection context
0810 22:33:46.832510 (+ 170us) client_negotiation.cc:240] Negotiation successful
0810 22:33:46.832796 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":153,"threads_started":1}
I20260810 22:33:46.835610 19461 heartbeater.cc:347] Connected to a master server at 127.16.233.126:43475
I20260810 22:33:46.835888 19461 heartbeater.cc:464] Registering TS with master...
I20260810 22:33:46.836462 19461 heartbeater.cc:511] Master 127.16.233.126:43475 requested a full tablet report, sending...
I20260810 22:33:46.838491 19224 ts_manager.cc:194] Registered new tserver with Master: 5d3b49686ca5430db55bc282694fb9d4 (127.16.233.65:43731)
I20260810 22:33:46.840329 19224 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.65@KRBTEST.COM'} at 127.16.233.65:57243
WARNING: no policy specified for kudu/127.16.233.66@KRBTEST.COM; defaulting to no policy
I20260810 22:33:46.848317 19281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:46.838577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:45929 (local address 127.16.233.126:43475)
0810 22:33:46.838705 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:46.838710 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:46.839659 (+ 949us) server_negotiation.cc:407] Connection header received
0810 22:33:46.839762 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:46.839768 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:46.839826 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:46.839919 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:46.840511 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:46.841457 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:46.842548 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:46.842813 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:46.842945 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:46.844345 (+ 1400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:46.844349 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:46.844352 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:46.844381 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:46.846248 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:46.846827 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:46.846831 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:46.846833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:46.846933 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:46.847291 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:46.847295 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:46.847298 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:46.847516 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:46.847647 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:46.847993 (+ 346us) server_negotiation.cc:299] Negotiation successful
0810 22:33:46.848127 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 22:33:46.848380 19469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:46.838537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:43475 (local address 127.16.233.65:45929)
0810 22:33:46.839538 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:46.839556 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:46.839675 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:46.839959 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:46.839962 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:46.840216 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:33:46.840363 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:46.840373 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:46.841623 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:46.841627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:46.842394 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:46.842404 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:46.842953 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:46.842957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:46.843059 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:46.843708 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:33:46.843722 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:33:46.844155 (+ 433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:33:46.846399 (+ 2244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:46.846403 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:46.846405 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:46.846654 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:46.847046 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:46.847050 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:46.847052 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:46.847156 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:46.847635 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:33:46.847638 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:33:46.847756 (+ 118us) client_negotiation.cc:769] Sending connection context
0810 22:33:46.847904 (+ 148us) client_negotiation.cc:240] Negotiation successful
0810 22:33:46.848097 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":901,"mutex_wait_us":42,"thread_start_us":108,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab.
Entry for principal kudu/127.16.233.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab.
I20260810 22:33:46.869419 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.66:0
--local_ip_for_outbound_sockets=127.16.233.66
--webserver_interface=127.16.233.66
--webserver_port=0
--tserver_master_addrs=127.16.233.126:43475
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab
--principal=kudu/127.16.233.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:34979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:33:46.989993 19471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:33:46.990427 19471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:33:46.990520 19471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:33:46.998337 19471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:33:46.998450 19471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:33:46.998528 19471 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 22:33:46.998561 19471 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 22:33:46.998629 19471 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.66
I20260810 22:33:47.004712 19471 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:34979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab
--principal=kudu/127.16.233.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.233.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.233.126:43475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.66
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:33:47.006011 19471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:33:47.007217 19471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:33:47.017968 19476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:33:47.018062 19477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:33:47.018024 19479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:33:47.018196 19471 server_base.cc:1034] running on GCE node
I20260810 22:33:47.018731 19471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:33:47.019479 19471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:33:47.020671 19471 hybrid_clock.cc:648] HybridClock initialized: now 1786401227020650 us; error 27 us; skew 500 ppm
Aug 10 22:33:47 dist-test-slave-77v9 krb5kdc[19178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401227, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:33:47.023795 19471 init.cc:377] Logged in from keytab as kudu/127.16.233.66@KRBTEST.COM (short username kudu)
I20260810 22:33:47.025002 19471 webserver.cc:460] Webserver started at http://127.16.233.66:35407/ using document root <none> and password file <none>
I20260810 22:33:47.025697 19471 fs_manager.cc:359] Metadata directory not provided
I20260810 22:33:47.025803 19471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:33:47.026057 19471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:33:47.028486 19471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/ts-1/data/instance:
uuid: "9eb2f1a7fb524990b88665f5acd19dbd"
format_stamp: "Formatted at 2026-08-10 22:33:47 on dist-test-slave-77v9"
server_key: "ffcaf2a76570f88bb6ad513ab8ccc47f"
server_key_iv: "de09ccc3f107043bc3b92a51781b6c14"
server_key_version: "encryptionkey@0"
I20260810 22:33:47.029328 19471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/ts-1/wal/instance:
uuid: "9eb2f1a7fb524990b88665f5acd19dbd"
format_stamp: "Formatted at 2026-08-10 22:33:47 on dist-test-slave-77v9"
server_key: "ffcaf2a76570f88bb6ad513ab8ccc47f"
server_key_iv: "de09ccc3f107043bc3b92a51781b6c14"
server_key_version: "encryptionkey@0"
I20260810 22:33:47.034068 19471 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 22:33:47.036988 19486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:47.038525 19471 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:33:47.038671 19471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/ts-1/wal
uuid: "9eb2f1a7fb524990b88665f5acd19dbd"
format_stamp: "Formatted at 2026-08-10 22:33:47 on dist-test-slave-77v9"
server_key: "ffcaf2a76570f88bb6ad513ab8ccc47f"
server_key_iv: "de09ccc3f107043bc3b92a51781b6c14"
server_key_version: "encryptionkey@0"
I20260810 22:33:47.038807 19471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:33:47.076241 19471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:33:47.079921 19471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:33:47.080163 19471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:33:47.081959 19471 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:33:47.082055 19471 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:47.082136 19471 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:33:47.082170 19471 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 22:33:47 dist-test-slave-77v9 krb5kdc[19178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401227, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.66@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:33:47.094317 19471 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.66:39973
I20260810 22:33:47.095454 19471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
I20260810 22:33:47.095729 19610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.66:39973 every 8 connection(s)
I20260810 22:33:47.097059 19281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:47.082835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:44685 (local address 127.16.233.126:43475)
0810 22:33:47.082981 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:47.082985 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:47.084203 (+ 1218us) server_negotiation.cc:407] Connection header received
0810 22:33:47.085494 (+ 1291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:47.085502 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:47.085573 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:47.085683 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:47.087163 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:47.087709 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:47.089066 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:47.089285 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:47.089401 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:47.092550 (+ 3149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:47.092554 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:47.092557 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:47.092594 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:47.094459 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:47.095192 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:47.095196 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:47.095197 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:47.095296 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:47.095712 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:47.095715 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:47.095717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:47.096002 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:47.096104 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:47.096739 (+ 635us) server_negotiation.cc:299] Negotiation successful
0810 22:33:47.096933 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 22:33:47.097908 19498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:47.083470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:43475 (local address 127.16.233.66:44685)
0810 22:33:47.084030 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:47.084079 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:47.085205 (+ 1126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:47.085890 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:47.085898 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:47.086433 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:33:47.087014 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:47.087038 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:47.087847 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:47.087851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:47.088925 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:47.088943 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:47.089461 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:47.089466 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:47.089651 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:47.090443 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:33:47.090471 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:33:47.092361 (+ 1890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:33:47.094596 (+ 2235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:47.094601 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:47.094616 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:47.095053 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:47.095409 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:47.095412 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:47.095414 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:47.095611 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:47.096201 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:33:47.096207 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:33:47.096496 (+ 289us) client_negotiation.cc:769] Sending connection context
0810 22:33:47.096656 (+ 160us) client_negotiation.cc:240] Negotiation successful
0810 22:33:47.097000 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":141,"threads_started":1}
I20260810 22:33:47.099006 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 19471
I20260810 22:33:47.099145 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/ts-1/wal/instance
I20260810 22:33:47.099314 19611 heartbeater.cc:347] Connected to a master server at 127.16.233.126:43475
I20260810 22:33:47.099457 19611 heartbeater.cc:464] Registering TS with master...
I20260810 22:33:47.099493 17317 external_mini_cluster.cc:1428] Setting key d5e0d88d4f5ad2a19c877b1092e6ee55
I20260810 22:33:47.099988 19611 heartbeater.cc:511] Master 127.16.233.126:43475 requested a full tablet report, sending...
I20260810 22:33:47.101086 19224 ts_manager.cc:194] Registered new tserver with Master: 9eb2f1a7fb524990b88665f5acd19dbd (127.16.233.66:39973)
I20260810 22:33:47.101728 19224 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.66@KRBTEST.COM'} at 127.16.233.66:44685
I20260810 22:33:47.109927 19281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:47.101997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:46489 (local address 127.16.233.126:43475)
0810 22:33:47.102141 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:47.102148 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:47.102461 (+ 313us) server_negotiation.cc:407] Connection header received
0810 22:33:47.102556 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:47.102561 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:47.102617 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:47.102710 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:47.103257 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:47.103933 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:47.104791 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:47.105057 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:47.105172 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:47.106400 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:47.106403 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:47.106406 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:47.106442 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:47.108055 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:47.108521 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:47.108525 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:47.108527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:47.108610 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:47.108986 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:47.108991 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:47.108993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:47.109238 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:47.109356 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:47.109635 (+ 279us) server_negotiation.cc:299] Negotiation successful
0810 22:33:47.109756 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 22:33:47.109951 19617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:47.102024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:43475 (local address 127.16.233.66:46489)
0810 22:33:47.102357 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:47.102372 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:47.102472 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:47.102751 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:47.102754 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:47.102996 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:33:47.103127 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:47.103135 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:47.104076 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:47.104079 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:47.104649 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:47.104657 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:47.105180 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:47.105184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:47.105283 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:47.105867 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:33:47.105879 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:33:47.106245 (+ 366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:33:47.108185 (+ 1940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:47.108189 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:47.108191 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:47.108404 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:47.108711 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:47.108715 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:47.108717 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:47.108801 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:47.109344 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:33:47.109347 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:33:47.109456 (+ 109us) client_negotiation.cc:769] Sending connection context
0810 22:33:47.109565 (+ 109us) client_negotiation.cc:240] Negotiation successful
0810 22:33:47.109743 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":99,"threads_started":1}
WARNING: no policy specified for kudu/127.16.233.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab.
Entry for principal kudu/127.16.233.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab.
I20260810 22:33:47.132911 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.67:0
--local_ip_for_outbound_sockets=127.16.233.67
--webserver_interface=127.16.233.67
--webserver_port=0
--tserver_master_addrs=127.16.233.126:43475
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab
--principal=kudu/127.16.233.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:34979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:33:47.256839 19620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:33:47.257174 19620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:33:47.257226 19620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:33:47.261600 19620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:33:47.261679 19620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:33:47.261722 19620 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 22:33:47.261737 19620 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 22:33:47.261771 19620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.67
I20260810 22:33:47.267356 19620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:34979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab
--principal=kudu/127.16.233.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.233.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.233.126:43475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19620
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.67
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:33:47.268570 19620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:33:47.269691 19620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:33:47.279973 19625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:33:47.280004 19628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:33:47.279996 19626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:33:47.280475 19620 server_base.cc:1034] running on GCE node
I20260810 22:33:47.280992 19620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:33:47.281737 19620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:33:47.282931 19620 hybrid_clock.cc:648] HybridClock initialized: now 1786401227282896 us; error 47 us; skew 500 ppm
Aug 10 22:33:47 dist-test-slave-77v9 krb5kdc[19178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401227, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:33:47.286109 19620 init.cc:377] Logged in from keytab as kudu/127.16.233.67@KRBTEST.COM (short username kudu)
I20260810 22:33:47.287173 19620 webserver.cc:460] Webserver started at http://127.16.233.67:40347/ using document root <none> and password file <none>
I20260810 22:33:47.287856 19620 fs_manager.cc:359] Metadata directory not provided
I20260810 22:33:47.287955 19620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:33:47.288179 19620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:33:47.290525 19620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/ts-2/data/instance:
uuid: "029932492fa8492c80cc896ab4c881f6"
format_stamp: "Formatted at 2026-08-10 22:33:47 on dist-test-slave-77v9"
server_key: "e8b8714364908f906a59ce7ce09b7449"
server_key_iv: "9486e4dfbe9347e4affa43283ff3fd05"
server_key_version: "encryptionkey@0"
I20260810 22:33:47.291128 19620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/ts-2/wal/instance:
uuid: "029932492fa8492c80cc896ab4c881f6"
format_stamp: "Formatted at 2026-08-10 22:33:47 on dist-test-slave-77v9"
server_key: "e8b8714364908f906a59ce7ce09b7449"
server_key_iv: "9486e4dfbe9347e4affa43283ff3fd05"
server_key_version: "encryptionkey@0"
I20260810 22:33:47.295449 19620 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 22:33:47.298233 19635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:47.299767 19620 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 22:33:47.299933 19620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/ts-2/wal
uuid: "029932492fa8492c80cc896ab4c881f6"
format_stamp: "Formatted at 2026-08-10 22:33:47 on dist-test-slave-77v9"
server_key: "e8b8714364908f906a59ce7ce09b7449"
server_key_iv: "9486e4dfbe9347e4affa43283ff3fd05"
server_key_version: "encryptionkey@0"
I20260810 22:33:47.300097 19620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:33:47.334584 19620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:33:47.338449 19620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:33:47.338682 19620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:33:47.340250 19620 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:33:47.340304 19620 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:47.340345 19620 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:33:47.340407 19620 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 22:33:47 dist-test-slave-77v9 krb5kdc[19178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401227, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.67@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:33:47.352591 19620 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.67:40807
I20260810 22:33:47.352659 19770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.67:40807 every 8 connection(s)
I20260810 22:33:47.353835 19620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
I20260810 22:33:47.357302 19281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:47.341327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:57371 (local address 127.16.233.126:43475)
0810 22:33:47.341479 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:47.341483 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:47.342972 (+ 1489us) server_negotiation.cc:407] Connection header received
0810 22:33:47.344333 (+ 1361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:47.344337 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:47.344401 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:47.344498 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:47.346380 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:47.347308 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:47.348923 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:47.349224 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:47.349345 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:47.352994 (+ 3649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:47.352997 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:47.353000 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:47.353040 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:47.354658 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:47.355385 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:47.355388 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:47.355389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:47.355486 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:47.355964 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:47.355968 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:47.355969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:47.356194 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:47.356291 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:47.357042 (+ 751us) server_negotiation.cc:299] Negotiation successful
0810 22:33:47.357169 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 22:33:47.358083 19645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:47.341924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:43475 (local address 127.16.233.67:57371)
0810 22:33:47.342698 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:47.342803 (+ 105us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:47.344104 (+ 1301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:47.344738 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:47.344748 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:47.345362 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:33:47.346167 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:47.346190 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:47.347460 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:47.347465 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:47.348742 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:47.348752 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:47.349410 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:47.349414 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:47.349645 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:47.350364 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:33:47.350384 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:33:47.352763 (+ 2379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:33:47.354832 (+ 2069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:47.354839 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:47.354854 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:47.355178 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:47.355578 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:47.355580 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:47.355581 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:47.355713 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:47.356317 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:33:47.356323 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:33:47.356567 (+ 244us) client_negotiation.cc:769] Sending connection context
0810 22:33:47.356725 (+ 158us) client_negotiation.cc:240] Negotiation successful
0810 22:33:47.357130 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":146,"threads_started":1}
I20260810 22:33:47.359504 19771 heartbeater.cc:347] Connected to a master server at 127.16.233.126:43475
I20260810 22:33:47.359709 19771 heartbeater.cc:464] Registering TS with master...
I20260810 22:33:47.360325 19771 heartbeater.cc:511] Master 127.16.233.126:43475 requested a full tablet report, sending...
I20260810 22:33:47.361568 19227 ts_manager.cc:194] Registered new tserver with Master: 029932492fa8492c80cc896ab4c881f6 (127.16.233.67:40807)
I20260810 22:33:47.362254 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 19620
I20260810 22:33:47.362353 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/ts-2/wal/instance
I20260810 22:33:47.362380 19227 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.67@KRBTEST.COM'} at 127.16.233.67:57371
I20260810 22:33:47.362728 17317 external_mini_cluster.cc:1428] Setting key c2925b694ebaa5ba4073e456cab15e63
I20260810 22:33:47.365111 17317 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 22:33:47.371490 19281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:47.361972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:59201 (local address 127.16.233.126:43475)
0810 22:33:47.362720 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:47.362724 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:47.363290 (+ 566us) server_negotiation.cc:407] Connection header received
0810 22:33:47.363465 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:47.363468 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:47.363517 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:47.363602 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:47.364093 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:47.364899 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:47.365806 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:47.366234 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:47.366372 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:47.367723 (+ 1351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:47.367730 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:47.367736 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:47.367771 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:47.369536 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:47.370089 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:47.370092 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:47.370094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:47.370177 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:47.370522 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:47.370524 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:47.370526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:47.370808 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:47.370902 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:47.371240 (+ 338us) server_negotiation.cc:299] Negotiation successful
0810 22:33:47.371358 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":685}
I20260810 22:33:47.371582 19777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:47.361876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:43475 (local address 127.16.233.67:59201)
0810 22:33:47.363204 (+ 1328us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:47.363219 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:47.363346 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:47.363620 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:47.363623 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:47.363833 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:33:47.363990 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:47.364000 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:47.365048 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:47.365051 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:47.365677 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:47.365685 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:47.366376 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:47.366379 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:47.366463 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:47.367119 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:33:47.367135 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:33:47.367556 (+ 421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:33:47.369664 (+ 2108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:47.369670 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:47.369676 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:47.369975 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:47.370288 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:47.370294 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:47.370300 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:47.370421 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:47.370928 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:33:47.370932 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:33:47.371046 (+ 114us) client_negotiation.cc:769] Sending connection context
0810 22:33:47.371176 (+ 130us) client_negotiation.cc:240] Negotiation successful
0810 22:33:47.371350 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1220,"thread_start_us":109,"threads_started":1}
Aug 10 22:33:47 dist-test-slave-77v9 krb5kdc[19178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 22:33:47 dist-test-slave-77v9 krb5kdc[19178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:33:47.396422 19281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:47.387318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34292 (local address 127.16.233.126:43475)
0810 22:33:47.387486 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:47.387495 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:47.387510 (+ 15us) server_negotiation.cc:407] Connection header received
0810 22:33:47.387555 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:47.387558 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:47.387600 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:47.387679 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:47.388467 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:47.389182 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:47.390080 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:47.390300 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:47.390384 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:47.392946 (+ 2562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:47.392950 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:47.392952 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:47.392979 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:47.394566 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:47.395042 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:47.395045 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:47.395046 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:47.395209 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:47.395550 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:47.395553 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:47.395554 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:47.395758 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:47.395876 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:47.396098 (+ 222us) server_negotiation.cc:299] Negotiation successful
0810 22:33:47.396204 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260810 22:33:47.399736 19227 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34292:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 22:33:47.403100 19227 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:33:47.420250 19791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:47.413013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.67:40807 (local address 127.0.0.1:53392)
0810 22:33:47.413534 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:47.413591 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:47.413732 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:47.414982 (+ 1250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:47.414991 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:47.415017 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:33:47.415243 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:47.415253 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:47.416998 (+ 1745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:47.417001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:47.418446 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:47.418458 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:47.419608 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:47.419612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:47.419769 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:47.419821 (+ 52us) client_negotiation.cc:769] Sending connection context
0810 22:33:47.419937 (+ 116us) client_negotiation.cc:240] Negotiation successful
0810 22:33:47.420106 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"mutex_wait_us":237,"thread_start_us":112,"threads_started":1}
I20260810 22:33:47.420297 19793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:47.413802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53392 (local address 127.16.233.67:40807)
0810 22:33:47.414377 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:47.414391 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:47.414417 (+ 26us) server_negotiation.cc:407] Connection header received
0810 22:33:47.414507 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:47.414514 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:47.414785 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:47.415445 (+ 660us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:33:47.415608 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:47.416872 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:47.418602 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:47.419460 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:47.419696 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:47.419896 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:47.419971 (+ 75us) server_negotiation.cc:299] Negotiation successful
0810 22:33:47.420119 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":190,"threads_started":1}
I20260810 22:33:47.420953 19794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:47.414150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36462 (local address 127.16.233.65:43731)
0810 22:33:47.414451 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:47.414458 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:47.414836 (+ 378us) server_negotiation.cc:407] Connection header received
0810 22:33:47.414971 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:47.414975 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:47.415157 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:47.415300 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:33:47.415646 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:47.416688 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:47.418614 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:47.419200 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:47.419311 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:47.419492 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:47.420670 (+ 1178us) server_negotiation.cc:299] Negotiation successful
0810 22:33:47.420783 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":99,"threads_started":1}
I20260810 22:33:47.421521 19792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:47.413079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.66:39973 (local address 127.0.0.1:58396)
0810 22:33:47.414531 (+ 1452us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:47.414546 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:47.414641 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:47.415123 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:47.415126 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:47.415144 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:33:47.415314 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:47.415321 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:47.417044 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:47.417048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:47.418112 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:47.418123 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:47.419594 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:47.419598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:47.420725 (+ 1127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:47.420845 (+ 120us) client_negotiation.cc:769] Sending connection context
0810 22:33:47.420984 (+ 139us) client_negotiation.cc:240] Negotiation successful
0810 22:33:47.421172 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1382,"mutex_wait_us":1011,"thread_start_us":139,"threads_started":1}
I20260810 22:33:47.422019 19795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:47.414324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58396 (local address 127.16.233.66:39973)
0810 22:33:47.414646 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:47.414658 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:47.414707 (+ 49us) server_negotiation.cc:407] Connection header received
0810 22:33:47.414780 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:47.414786 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:47.414980 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:47.415145 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:33:47.415446 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:47.416871 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:47.418285 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:47.419175 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:47.419315 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:47.419517 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:47.421779 (+ 2262us) server_negotiation.cc:299] Negotiation successful
0810 22:33:47.421900 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":130,"threads_started":1}
I20260810 22:33:47.422955 19790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:47.413805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.65:43731 (local address 127.0.0.1:36462)
0810 22:33:47.414716 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:47.414730 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:47.414842 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:47.415449 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:47.415452 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:47.415468 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:33:47.415531 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:47.415535 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:47.416837 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:47.416843 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:47.418446 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:47.418458 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:47.419556 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:47.419560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:47.420485 (+ 925us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:47.420521 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 22:33:47.421142 (+ 621us) client_negotiation.cc:240] Negotiation successful
0810 22:33:47.422850 (+ 1708us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":845,"mutex_wait_us":841,"thread_start_us":117,"threads_started":1}
I20260810 22:33:47.423646 19692 tablet_service.cc:1467] Processing CreateTablet for tablet 797b6cafc2c54861aeef1c1a0a3a1a80 (DEFAULT_TABLE table=test-table [id=8845afa1dada467689ab4949b3ff822a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:33:47.424930 19692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 797b6cafc2c54861aeef1c1a0a3a1a80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:33:47.424661 19542 tablet_service.cc:1467] Processing CreateTablet for tablet 797b6cafc2c54861aeef1c1a0a3a1a80 (DEFAULT_TABLE table=test-table [id=8845afa1dada467689ab4949b3ff822a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:33:47.425793 19542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 797b6cafc2c54861aeef1c1a0a3a1a80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:33:47.425822 19381 tablet_service.cc:1467] Processing CreateTablet for tablet 797b6cafc2c54861aeef1c1a0a3a1a80 (DEFAULT_TABLE table=test-table [id=8845afa1dada467689ab4949b3ff822a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:33:47.426903 19381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 797b6cafc2c54861aeef1c1a0a3a1a80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:33:47.435833 19796 tablet_bootstrap.cc:492] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 5d3b49686ca5430db55bc282694fb9d4: Bootstrap starting.
I20260810 22:33:47.435822 19798 tablet_bootstrap.cc:492] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 9eb2f1a7fb524990b88665f5acd19dbd: Bootstrap starting.
I20260810 22:33:47.438093 19796 tablet_bootstrap.cc:654] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 5d3b49686ca5430db55bc282694fb9d4: Neither blocks nor log segments found. Creating new log.
I20260810 22:33:47.438202 19798 tablet_bootstrap.cc:654] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 9eb2f1a7fb524990b88665f5acd19dbd: Neither blocks nor log segments found. Creating new log.
I20260810 22:33:47.438999 19796 log.cc:824] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 5d3b49686ca5430db55bc282694fb9d4: Log is configured to *not* fsync() on all Append() calls
I20260810 22:33:47.439028 19798 log.cc:824] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 9eb2f1a7fb524990b88665f5acd19dbd: Log is configured to *not* fsync() on all Append() calls
I20260810 22:33:47.439392 19799 tablet_bootstrap.cc:492] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 029932492fa8492c80cc896ab4c881f6: Bootstrap starting.
I20260810 22:33:47.441190 19796 tablet_bootstrap.cc:492] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 5d3b49686ca5430db55bc282694fb9d4: No bootstrap required, opened a new log
I20260810 22:33:47.441424 19796 ts_tablet_manager.cc:1397] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 5d3b49686ca5430db55bc282694fb9d4: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 22:33:47.441586 19798 tablet_bootstrap.cc:492] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 9eb2f1a7fb524990b88665f5acd19dbd: No bootstrap required, opened a new log
I20260810 22:33:47.441782 19798 ts_tablet_manager.cc:1397] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 9eb2f1a7fb524990b88665f5acd19dbd: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260810 22:33:47.442088 19799 tablet_bootstrap.cc:654] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 029932492fa8492c80cc896ab4c881f6: Neither blocks nor log segments found. Creating new log.
I20260810 22:33:47.443112 19799 log.cc:824] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 029932492fa8492c80cc896ab4c881f6: Log is configured to *not* fsync() on all Append() calls
I20260810 22:33:47.445438 19799 tablet_bootstrap.cc:492] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 029932492fa8492c80cc896ab4c881f6: No bootstrap required, opened a new log
I20260810 22:33:47.445640 19799 ts_tablet_manager.cc:1397] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 029932492fa8492c80cc896ab4c881f6: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260810 22:33:47.445781 19798 raft_consensus.cc:348] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 9eb2f1a7fb524990b88665f5acd19dbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "029932492fa8492c80cc896ab4c881f6" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 40807 } } peers { permanent_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39973 } } peers { permanent_uuid: "5d3b49686ca5430db55bc282694fb9d4" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 43731 } }
I20260810 22:33:47.446010 19798 raft_consensus.cc:374] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 9eb2f1a7fb524990b88665f5acd19dbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:33:47.446044 19798 raft_consensus.cc:729] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 9eb2f1a7fb524990b88665f5acd19dbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9eb2f1a7fb524990b88665f5acd19dbd, State: Initialized, Role: FOLLOWER
I20260810 22:33:47.446173 19796 raft_consensus.cc:348] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 5d3b49686ca5430db55bc282694fb9d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "029932492fa8492c80cc896ab4c881f6" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 40807 } } peers { permanent_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39973 } } peers { permanent_uuid: "5d3b49686ca5430db55bc282694fb9d4" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 43731 } }
I20260810 22:33:47.446519 19796 raft_consensus.cc:374] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 5d3b49686ca5430db55bc282694fb9d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:33:47.446640 19796 raft_consensus.cc:729] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 5d3b49686ca5430db55bc282694fb9d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d3b49686ca5430db55bc282694fb9d4, State: Initialized, Role: FOLLOWER
I20260810 22:33:47.446557 19798 consensus_queue.cc:260] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 9eb2f1a7fb524990b88665f5acd19dbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "029932492fa8492c80cc896ab4c881f6" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 40807 } } peers { permanent_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39973 } } peers { permanent_uuid: "5d3b49686ca5430db55bc282694fb9d4" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 43731 } }
I20260810 22:33:47.447088 19796 consensus_queue.cc:260] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 5d3b49686ca5430db55bc282694fb9d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "029932492fa8492c80cc896ab4c881f6" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 40807 } } peers { permanent_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39973 } } peers { permanent_uuid: "5d3b49686ca5430db55bc282694fb9d4" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 43731 } }
I20260810 22:33:47.447479 19611 heartbeater.cc:503] Master 127.16.233.126:43475 was elected leader, sending a full tablet report...
I20260810 22:33:47.447842 19798 ts_tablet_manager.cc:1428] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 9eb2f1a7fb524990b88665f5acd19dbd: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 22:33:47.448204 19796 ts_tablet_manager.cc:1428] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 5d3b49686ca5430db55bc282694fb9d4: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260810 22:33:47.448596 19461 heartbeater.cc:503] Master 127.16.233.126:43475 was elected leader, sending a full tablet report...
I20260810 22:33:47.450085 19799 raft_consensus.cc:348] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 029932492fa8492c80cc896ab4c881f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "029932492fa8492c80cc896ab4c881f6" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 40807 } } peers { permanent_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39973 } } peers { permanent_uuid: "5d3b49686ca5430db55bc282694fb9d4" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 43731 } }
I20260810 22:33:47.450394 19799 raft_consensus.cc:374] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 029932492fa8492c80cc896ab4c881f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:33:47.450460 19799 raft_consensus.cc:729] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 029932492fa8492c80cc896ab4c881f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 029932492fa8492c80cc896ab4c881f6, State: Initialized, Role: FOLLOWER
I20260810 22:33:47.450896 19799 consensus_queue.cc:260] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 029932492fa8492c80cc896ab4c881f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "029932492fa8492c80cc896ab4c881f6" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 40807 } } peers { permanent_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39973 } } peers { permanent_uuid: "5d3b49686ca5430db55bc282694fb9d4" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 43731 } }
I20260810 22:33:47.451649 19771 heartbeater.cc:503] Master 127.16.233.126:43475 was elected leader, sending a full tablet report...
I20260810 22:33:47.451967 19799 ts_tablet_manager.cc:1428] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 029932492fa8492c80cc896ab4c881f6: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260810 22:33:47.581912 19462 tablet.cc:2369] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 5d3b49686ca5430db55bc282694fb9d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 22:33:47.596856 19612 tablet.cc:2369] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 9eb2f1a7fb524990b88665f5acd19dbd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 22:33:47.605063 19772 tablet.cc:2369] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 029932492fa8492c80cc896ab4c881f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:33:47.698797 19806 raft_consensus.cc:482] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 029932492fa8492c80cc896ab4c881f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:33:47.699122 19806 raft_consensus.cc:504] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 029932492fa8492c80cc896ab4c881f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "029932492fa8492c80cc896ab4c881f6" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 40807 } } peers { permanent_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39973 } } peers { permanent_uuid: "5d3b49686ca5430db55bc282694fb9d4" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 43731 } }
I20260810 22:33:47.700496 19806 leader_election.cc:288] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 029932492fa8492c80cc896ab4c881f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9eb2f1a7fb524990b88665f5acd19dbd (127.16.233.66:39973), 5d3b49686ca5430db55bc282694fb9d4 (127.16.233.65:43731)
I20260810 22:33:47.704210 19645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:47.700732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.66:39973 (local address 127.16.233.67:49003)
0810 22:33:47.700891 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:47.700909 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:47.701016 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:47.701353 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:47.701356 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:47.701375 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:33:47.701512 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:47.701518 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:47.702284 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:47.702287 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:47.703092 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:47.703100 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:47.703763 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:47.703766 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:47.703882 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:47.703913 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 22:33:47.703993 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 22:33:47.704071 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"spinlock_wait_cycles":27008}
I20260810 22:33:47.704210 19795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:47.700986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:49003 (local address 127.16.233.66:39973)
0810 22:33:47.701128 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:47.701132 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:47.701153 (+ 21us) server_negotiation.cc:407] Connection header received
0810 22:33:47.701200 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:47.701202 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:47.701239 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:47.701339 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:33:47.701629 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:47.702152 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:47.703236 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:47.703635 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:47.703778 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:47.703838 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:47.704014 (+ 176us) server_negotiation.cc:299] Negotiation successful
0810 22:33:47.704084 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 22:33:47.705117 19807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:47.700805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.65:43731 (local address 127.16.233.67:56051)
0810 22:33:47.701253 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:47.701265 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:47.701414 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:47.701696 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:47.701699 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:47.701714 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:33:47.702223 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:47.702231 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:47.702970 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:47.702976 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:47.703735 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:47.703744 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:47.704643 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:47.704647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:47.704751 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:47.704769 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 22:33:47.704908 (+ 139us) client_negotiation.cc:240] Negotiation successful
0810 22:33:47.704973 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"spinlock_wait_cycles":768,"thread_start_us":118,"threads_started":1}
I20260810 22:33:47.705117 19794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:47.700961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:56051 (local address 127.16.233.65:43731)
0810 22:33:47.701108 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:47.701114 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:47.701363 (+ 249us) server_negotiation.cc:407] Connection header received
0810 22:33:47.701539 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:47.701542 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:47.701585 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:47.701686 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:33:47.702354 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:47.702854 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:47.703868 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:47.704515 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:47.704634 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:47.704687 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:47.704930 (+ 243us) server_negotiation.cc:299] Negotiation successful
0810 22:33:47.705010 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 22:33:47.705072 19563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "797b6cafc2c54861aeef1c1a0a3a1a80" candidate_uuid: "029932492fa8492c80cc896ab4c881f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" is_pre_election: true
I20260810 22:33:47.705446 19563 raft_consensus.cc:2454] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 9eb2f1a7fb524990b88665f5acd19dbd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 029932492fa8492c80cc896ab4c881f6 in term 0.
I20260810 22:33:47.705914 19404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "797b6cafc2c54861aeef1c1a0a3a1a80" candidate_uuid: "029932492fa8492c80cc896ab4c881f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d3b49686ca5430db55bc282694fb9d4" is_pre_election: true
I20260810 22:33:47.706022 19639 leader_election.cc:302] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 029932492fa8492c80cc896ab4c881f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 029932492fa8492c80cc896ab4c881f6, 9eb2f1a7fb524990b88665f5acd19dbd; no voters:
I20260810 22:33:47.706216 19404 raft_consensus.cc:2454] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 5d3b49686ca5430db55bc282694fb9d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 029932492fa8492c80cc896ab4c881f6 in term 0.
I20260810 22:33:47.706419 19806 raft_consensus.cc:2781] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 029932492fa8492c80cc896ab4c881f6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:33:47.706542 19806 raft_consensus.cc:482] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 029932492fa8492c80cc896ab4c881f6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:33:47.706606 19806 raft_consensus.cc:3037] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 029932492fa8492c80cc896ab4c881f6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:33:47.707715 19806 raft_consensus.cc:504] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 029932492fa8492c80cc896ab4c881f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "029932492fa8492c80cc896ab4c881f6" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 40807 } } peers { permanent_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39973 } } peers { permanent_uuid: "5d3b49686ca5430db55bc282694fb9d4" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 43731 } }
I20260810 22:33:47.708107 19806 leader_election.cc:288] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 029932492fa8492c80cc896ab4c881f6 [CANDIDATE]: Term 1 election: Requested vote from peers 9eb2f1a7fb524990b88665f5acd19dbd (127.16.233.66:39973), 5d3b49686ca5430db55bc282694fb9d4 (127.16.233.65:43731)
I20260810 22:33:47.708499 19563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "797b6cafc2c54861aeef1c1a0a3a1a80" candidate_uuid: "029932492fa8492c80cc896ab4c881f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9eb2f1a7fb524990b88665f5acd19dbd"
I20260810 22:33:47.708700 19404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "797b6cafc2c54861aeef1c1a0a3a1a80" candidate_uuid: "029932492fa8492c80cc896ab4c881f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d3b49686ca5430db55bc282694fb9d4"
I20260810 22:33:47.708705 19563 raft_consensus.cc:3037] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 9eb2f1a7fb524990b88665f5acd19dbd [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:33:47.708897 19404 raft_consensus.cc:3037] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 5d3b49686ca5430db55bc282694fb9d4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:33:47.710003 19563 raft_consensus.cc:2454] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 9eb2f1a7fb524990b88665f5acd19dbd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 029932492fa8492c80cc896ab4c881f6 in term 1.
I20260810 22:33:47.710165 19404 raft_consensus.cc:2454] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 5d3b49686ca5430db55bc282694fb9d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 029932492fa8492c80cc896ab4c881f6 in term 1.
I20260810 22:33:47.710450 19639 leader_election.cc:302] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 029932492fa8492c80cc896ab4c881f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 029932492fa8492c80cc896ab4c881f6, 9eb2f1a7fb524990b88665f5acd19dbd; no voters:
I20260810 22:33:47.710655 19806 raft_consensus.cc:2781] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 029932492fa8492c80cc896ab4c881f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:33:47.710919 19806 raft_consensus.cc:686] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 029932492fa8492c80cc896ab4c881f6 [term 1 LEADER]: Becoming Leader. State: Replica: 029932492fa8492c80cc896ab4c881f6, State: Running, Role: LEADER
I20260810 22:33:47.711300 19806 consensus_queue.cc:237] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 029932492fa8492c80cc896ab4c881f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "029932492fa8492c80cc896ab4c881f6" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 40807 } } peers { permanent_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39973 } } peers { permanent_uuid: "5d3b49686ca5430db55bc282694fb9d4" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 43731 } }
I20260810 22:33:47.714772 19224 catalog_manager.cc:5495] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 029932492fa8492c80cc896ab4c881f6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 029932492fa8492c80cc896ab4c881f6 (127.16.233.67). New cstate: current_term: 1 leader_uuid: "029932492fa8492c80cc896ab4c881f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "029932492fa8492c80cc896ab4c881f6" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 40807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d3b49686ca5430db55bc282694fb9d4" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 43731 } health_report { overall_health: UNKNOWN } } }
Aug 10 22:33:47 dist-test-slave-77v9 krb5kdc[19178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401226, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.126@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:33:47.820641 19816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:47.811008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:43475 (local address 127.0.0.1:34298)
0810 22:33:47.811341 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:47.811355 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:47.811521 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:47.811809 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:47.811812 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:47.812292 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:33:47.812393 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:47.812401 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:47.813361 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:47.813368 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:47.814000 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:47.814009 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:47.814567 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:47.814571 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:47.814668 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:47.815340 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:33:47.815353 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:33:47.817213 (+ 1860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:33:47.818998 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:47.819008 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:47.819014 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:47.819277 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:47.819602 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:47.819604 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:47.819605 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:47.819683 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:47.820115 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:33:47.820120 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:33:47.820232 (+ 112us) client_negotiation.cc:769] Sending connection context
0810 22:33:47.820328 (+ 96us) client_negotiation.cc:240] Negotiation successful
0810 22:33:47.820485 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":103,"threads_started":1}
I20260810 22:33:47.820873 19281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:47.811046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34298 (local address 127.16.233.126:43475)
0810 22:33:47.811268 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:47.811274 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:47.811450 (+ 176us) server_negotiation.cc:407] Connection header received
0810 22:33:47.811631 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:47.811634 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:47.811686 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:47.811788 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:47.812538 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:47.813240 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:47.814162 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:47.814422 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:47.814545 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:47.817370 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:47.817374 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:47.817376 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:47.817412 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:47.818855 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:47.819387 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:47.819390 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:47.819391 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:47.819497 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:47.819792 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:47.819795 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:47.819796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:47.820009 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:47.820091 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:47.820396 (+ 305us) server_negotiation.cc:299] Negotiation successful
0810 22:33:47.820585 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":90}
I20260810 22:33:47.825343 19224 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.16.233.126@KRBTEST.COM'} at 127.0.0.1:34298:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 22:33:47.826180 19224 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:33:47.831946 19381 tablet_service.cc:1467] Processing CreateTablet for tablet d1c46c238ae240208afe8f3986bffe44 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=41b0bbcd824249e5929a3f7ae6327a6a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 22:33:47.832005 19542 tablet_service.cc:1467] Processing CreateTablet for tablet d1c46c238ae240208afe8f3986bffe44 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=41b0bbcd824249e5929a3f7ae6327a6a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 22:33:47.832748 19542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1c46c238ae240208afe8f3986bffe44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:33:47.832721 19381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1c46c238ae240208afe8f3986bffe44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:33:47.833542 19692 tablet_service.cc:1467] Processing CreateTablet for tablet d1c46c238ae240208afe8f3986bffe44 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=41b0bbcd824249e5929a3f7ae6327a6a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 22:33:47.834375 19692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1c46c238ae240208afe8f3986bffe44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:33:47.835927 19798 tablet_bootstrap.cc:492] T d1c46c238ae240208afe8f3986bffe44 P 9eb2f1a7fb524990b88665f5acd19dbd: Bootstrap starting.
I20260810 22:33:47.836862 19796 tablet_bootstrap.cc:492] T d1c46c238ae240208afe8f3986bffe44 P 5d3b49686ca5430db55bc282694fb9d4: Bootstrap starting.
I20260810 22:33:47.836928 19798 tablet_bootstrap.cc:654] T d1c46c238ae240208afe8f3986bffe44 P 9eb2f1a7fb524990b88665f5acd19dbd: Neither blocks nor log segments found. Creating new log.
I20260810 22:33:47.837548 19799 tablet_bootstrap.cc:492] T d1c46c238ae240208afe8f3986bffe44 P 029932492fa8492c80cc896ab4c881f6: Bootstrap starting.
I20260810 22:33:47.837893 19796 tablet_bootstrap.cc:654] T d1c46c238ae240208afe8f3986bffe44 P 5d3b49686ca5430db55bc282694fb9d4: Neither blocks nor log segments found. Creating new log.
I20260810 22:33:47.838487 19799 tablet_bootstrap.cc:654] T d1c46c238ae240208afe8f3986bffe44 P 029932492fa8492c80cc896ab4c881f6: Neither blocks nor log segments found. Creating new log.
I20260810 22:33:47.839288 19796 tablet_bootstrap.cc:492] T d1c46c238ae240208afe8f3986bffe44 P 5d3b49686ca5430db55bc282694fb9d4: No bootstrap required, opened a new log
I20260810 22:33:47.839394 19796 ts_tablet_manager.cc:1397] T d1c46c238ae240208afe8f3986bffe44 P 5d3b49686ca5430db55bc282694fb9d4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 22:33:47.839946 19796 raft_consensus.cc:348] T d1c46c238ae240208afe8f3986bffe44 P 5d3b49686ca5430db55bc282694fb9d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3b49686ca5430db55bc282694fb9d4" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 43731 } } peers { permanent_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39973 } } peers { permanent_uuid: "029932492fa8492c80cc896ab4c881f6" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 40807 } }
I20260810 22:33:47.840080 19796 raft_consensus.cc:374] T d1c46c238ae240208afe8f3986bffe44 P 5d3b49686ca5430db55bc282694fb9d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:33:47.840056 19799 tablet_bootstrap.cc:492] T d1c46c238ae240208afe8f3986bffe44 P 029932492fa8492c80cc896ab4c881f6: No bootstrap required, opened a new log
I20260810 22:33:47.840117 19796 raft_consensus.cc:729] T d1c46c238ae240208afe8f3986bffe44 P 5d3b49686ca5430db55bc282694fb9d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d3b49686ca5430db55bc282694fb9d4, State: Initialized, Role: FOLLOWER
I20260810 22:33:47.840178 19799 ts_tablet_manager.cc:1397] T d1c46c238ae240208afe8f3986bffe44 P 029932492fa8492c80cc896ab4c881f6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 22:33:47.840179 19798 tablet_bootstrap.cc:492] T d1c46c238ae240208afe8f3986bffe44 P 9eb2f1a7fb524990b88665f5acd19dbd: No bootstrap required, opened a new log
I20260810 22:33:47.840261 19798 ts_tablet_manager.cc:1397] T d1c46c238ae240208afe8f3986bffe44 P 9eb2f1a7fb524990b88665f5acd19dbd: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20260810 22:33:47.840242 19796 consensus_queue.cc:260] T d1c46c238ae240208afe8f3986bffe44 P 5d3b49686ca5430db55bc282694fb9d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3b49686ca5430db55bc282694fb9d4" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 43731 } } peers { permanent_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39973 } } peers { permanent_uuid: "029932492fa8492c80cc896ab4c881f6" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 40807 } }
I20260810 22:33:47.840691 19799 raft_consensus.cc:348] T d1c46c238ae240208afe8f3986bffe44 P 029932492fa8492c80cc896ab4c881f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3b49686ca5430db55bc282694fb9d4" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 43731 } } peers { permanent_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39973 } } peers { permanent_uuid: "029932492fa8492c80cc896ab4c881f6" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 40807 } }
I20260810 22:33:47.840808 19799 raft_consensus.cc:374] T d1c46c238ae240208afe8f3986bffe44 P 029932492fa8492c80cc896ab4c881f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:33:47.840741 19798 raft_consensus.cc:348] T d1c46c238ae240208afe8f3986bffe44 P 9eb2f1a7fb524990b88665f5acd19dbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3b49686ca5430db55bc282694fb9d4" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 43731 } } peers { permanent_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39973 } } peers { permanent_uuid: "029932492fa8492c80cc896ab4c881f6" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 40807 } }
I20260810 22:33:47.840873 19799 raft_consensus.cc:729] T d1c46c238ae240208afe8f3986bffe44 P 029932492fa8492c80cc896ab4c881f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 029932492fa8492c80cc896ab4c881f6, State: Initialized, Role: FOLLOWER
I20260810 22:33:47.840911 19798 raft_consensus.cc:374] T d1c46c238ae240208afe8f3986bffe44 P 9eb2f1a7fb524990b88665f5acd19dbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:33:47.840952 19798 raft_consensus.cc:729] T d1c46c238ae240208afe8f3986bffe44 P 9eb2f1a7fb524990b88665f5acd19dbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9eb2f1a7fb524990b88665f5acd19dbd, State: Initialized, Role: FOLLOWER
I20260810 22:33:47.841022 19799 consensus_queue.cc:260] T d1c46c238ae240208afe8f3986bffe44 P 029932492fa8492c80cc896ab4c881f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3b49686ca5430db55bc282694fb9d4" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 43731 } } peers { permanent_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39973 } } peers { permanent_uuid: "029932492fa8492c80cc896ab4c881f6" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 40807 } }
I20260810 22:33:47.841131 19798 consensus_queue.cc:260] T d1c46c238ae240208afe8f3986bffe44 P 9eb2f1a7fb524990b88665f5acd19dbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3b49686ca5430db55bc282694fb9d4" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 43731 } } peers { permanent_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39973 } } peers { permanent_uuid: "029932492fa8492c80cc896ab4c881f6" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 40807 } }
I20260810 22:33:47.841360 19799 ts_tablet_manager.cc:1428] T d1c46c238ae240208afe8f3986bffe44 P 029932492fa8492c80cc896ab4c881f6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 22:33:47.841486 19798 ts_tablet_manager.cc:1428] T d1c46c238ae240208afe8f3986bffe44 P 9eb2f1a7fb524990b88665f5acd19dbd: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260810 22:33:47.840678 19805 tablet_replica.cc:439] T d1c46c238ae240208afe8f3986bffe44 P 5d3b49686ca5430db55bc282694fb9d4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3b49686ca5430db55bc282694fb9d4" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 43731 } } peers { permanent_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39973 } } peers { permanent_uuid: "029932492fa8492c80cc896ab4c881f6" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 40807 } } }
I20260810 22:33:47.841594 19806 tablet_replica.cc:439] T d1c46c238ae240208afe8f3986bffe44 P 029932492fa8492c80cc896ab4c881f6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3b49686ca5430db55bc282694fb9d4" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 43731 } } peers { permanent_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39973 } } peers { permanent_uuid: "029932492fa8492c80cc896ab4c881f6" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 40807 } } }
I20260810 22:33:47.841709 19806 tablet_replica.cc:442] T d1c46c238ae240208afe8f3986bffe44 P 029932492fa8492c80cc896ab4c881f6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 22:33:47.841712 19805 tablet_replica.cc:442] T d1c46c238ae240208afe8f3986bffe44 P 5d3b49686ca5430db55bc282694fb9d4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 22:33:47.841791 19804 tablet_replica.cc:439] T d1c46c238ae240208afe8f3986bffe44 P 9eb2f1a7fb524990b88665f5acd19dbd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3b49686ca5430db55bc282694fb9d4" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 43731 } } peers { permanent_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39973 } } peers { permanent_uuid: "029932492fa8492c80cc896ab4c881f6" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 40807 } } }
I20260810 22:33:47.841912 19804 tablet_replica.cc:442] T d1c46c238ae240208afe8f3986bffe44 P 9eb2f1a7fb524990b88665f5acd19dbd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 22:33:47.842792 19796 ts_tablet_manager.cc:1428] T d1c46c238ae240208afe8f3986bffe44 P 5d3b49686ca5430db55bc282694fb9d4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260810 22:33:47.887976 19804 raft_consensus.cc:482] T d1c46c238ae240208afe8f3986bffe44 P 9eb2f1a7fb524990b88665f5acd19dbd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:33:47.888147 19804 raft_consensus.cc:504] T d1c46c238ae240208afe8f3986bffe44 P 9eb2f1a7fb524990b88665f5acd19dbd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3b49686ca5430db55bc282694fb9d4" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 43731 } } peers { permanent_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39973 } } peers { permanent_uuid: "029932492fa8492c80cc896ab4c881f6" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 40807 } }
I20260810 22:33:47.889420 19804 leader_election.cc:288] T d1c46c238ae240208afe8f3986bffe44 P 9eb2f1a7fb524990b88665f5acd19dbd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d3b49686ca5430db55bc282694fb9d4 (127.16.233.65:43731), 029932492fa8492c80cc896ab4c881f6 (127.16.233.67:40807)
I20260810 22:33:47.893708 19794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:47.890004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:59751 (local address 127.16.233.65:43731)
0810 22:33:47.890160 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:47.890164 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:47.890285 (+ 121us) server_negotiation.cc:407] Connection header received
0810 22:33:47.890475 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:47.890481 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:47.890542 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:47.890595 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:33:47.890962 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:47.891483 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:47.892569 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:47.893085 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:47.893230 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:47.893296 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:47.893507 (+ 211us) server_negotiation.cc:299] Negotiation successful
0810 22:33:47.893582 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 22:33:47.893723 19820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:47.889865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.65:43731 (local address 127.16.233.66:59751)
0810 22:33:47.890163 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:47.890181 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:47.890311 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:47.890724 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:47.890726 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:47.890742 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:33:47.890847 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:47.890851 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:47.891613 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:47.891618 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:47.892434 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:47.892443 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:47.893249 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:47.893253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:47.893371 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:47.893391 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 22:33:47.893486 (+ 95us) client_negotiation.cc:240] Negotiation successful
0810 22:33:47.893560 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":127,"threads_started":1}
I20260810 22:33:47.894397 19404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1c46c238ae240208afe8f3986bffe44" candidate_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d3b49686ca5430db55bc282694fb9d4" is_pre_election: true
I20260810 22:33:47.894552 19821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:47.890290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.67:40807 (local address 127.16.233.66:42707)
0810 22:33:47.890586 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:47.890603 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:47.890802 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:47.891118 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:47.891121 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:47.891138 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:33:47.891634 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:47.891642 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:47.892441 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:47.892448 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:47.893244 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:47.893254 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:47.894144 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:47.894148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:47.894270 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:47.894290 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 22:33:47.894353 (+ 63us) client_negotiation.cc:240] Negotiation successful
0810 22:33:47.894401 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":85,"threads_started":1}
I20260810 22:33:47.894573 19404 raft_consensus.cc:2454] T d1c46c238ae240208afe8f3986bffe44 P 5d3b49686ca5430db55bc282694fb9d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9eb2f1a7fb524990b88665f5acd19dbd in term 0.
I20260810 22:33:47.894670 19793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:47.890452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:42707 (local address 127.16.233.67:40807)
0810 22:33:47.890603 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:47.890609 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:47.890676 (+ 67us) server_negotiation.cc:407] Connection header received
0810 22:33:47.890951 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:47.890954 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:47.891009 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:47.891105 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:33:47.891780 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:47.892310 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:47.893386 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:47.893867 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:47.894037 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:47.894101 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:47.894437 (+ 336us) server_negotiation.cc:299] Negotiation successful
0810 22:33:47.894542 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 22:33:47.894948 19493 leader_election.cc:302] T d1c46c238ae240208afe8f3986bffe44 P 9eb2f1a7fb524990b88665f5acd19dbd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d3b49686ca5430db55bc282694fb9d4, 9eb2f1a7fb524990b88665f5acd19dbd; no voters:
I20260810 22:33:47.895323 19804 raft_consensus.cc:2781] T d1c46c238ae240208afe8f3986bffe44 P 9eb2f1a7fb524990b88665f5acd19dbd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:33:47.895247 19714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1c46c238ae240208afe8f3986bffe44" candidate_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "029932492fa8492c80cc896ab4c881f6" is_pre_election: true
I20260810 22:33:47.895426 19804 raft_consensus.cc:482] T d1c46c238ae240208afe8f3986bffe44 P 9eb2f1a7fb524990b88665f5acd19dbd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:33:47.895529 19804 raft_consensus.cc:3037] T d1c46c238ae240208afe8f3986bffe44 P 9eb2f1a7fb524990b88665f5acd19dbd [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:33:47.895576 19714 raft_consensus.cc:2454] T d1c46c238ae240208afe8f3986bffe44 P 029932492fa8492c80cc896ab4c881f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9eb2f1a7fb524990b88665f5acd19dbd in term 0.
I20260810 22:33:47.896727 19804 raft_consensus.cc:504] T d1c46c238ae240208afe8f3986bffe44 P 9eb2f1a7fb524990b88665f5acd19dbd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3b49686ca5430db55bc282694fb9d4" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 43731 } } peers { permanent_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39973 } } peers { permanent_uuid: "029932492fa8492c80cc896ab4c881f6" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 40807 } }
I20260810 22:33:47.897123 19804 leader_election.cc:288] T d1c46c238ae240208afe8f3986bffe44 P 9eb2f1a7fb524990b88665f5acd19dbd [CANDIDATE]: Term 1 election: Requested vote from peers 5d3b49686ca5430db55bc282694fb9d4 (127.16.233.65:43731), 029932492fa8492c80cc896ab4c881f6 (127.16.233.67:40807)
I20260810 22:33:47.897586 19404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1c46c238ae240208afe8f3986bffe44" candidate_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d3b49686ca5430db55bc282694fb9d4"
I20260810 22:33:47.897627 19714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1c46c238ae240208afe8f3986bffe44" candidate_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "029932492fa8492c80cc896ab4c881f6"
I20260810 22:33:47.897744 19714 raft_consensus.cc:3037] T d1c46c238ae240208afe8f3986bffe44 P 029932492fa8492c80cc896ab4c881f6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:33:47.897739 19404 raft_consensus.cc:3037] T d1c46c238ae240208afe8f3986bffe44 P 5d3b49686ca5430db55bc282694fb9d4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:33:47.899075 19404 raft_consensus.cc:2454] T d1c46c238ae240208afe8f3986bffe44 P 5d3b49686ca5430db55bc282694fb9d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9eb2f1a7fb524990b88665f5acd19dbd in term 1.
I20260810 22:33:47.899138 19714 raft_consensus.cc:2454] T d1c46c238ae240208afe8f3986bffe44 P 029932492fa8492c80cc896ab4c881f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9eb2f1a7fb524990b88665f5acd19dbd in term 1.
I20260810 22:33:47.899443 19493 leader_election.cc:302] T d1c46c238ae240208afe8f3986bffe44 P 9eb2f1a7fb524990b88665f5acd19dbd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 029932492fa8492c80cc896ab4c881f6, 9eb2f1a7fb524990b88665f5acd19dbd; no voters:
I20260810 22:33:47.899680 19804 raft_consensus.cc:2781] T d1c46c238ae240208afe8f3986bffe44 P 9eb2f1a7fb524990b88665f5acd19dbd [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:33:47.899915 19804 raft_consensus.cc:686] T d1c46c238ae240208afe8f3986bffe44 P 9eb2f1a7fb524990b88665f5acd19dbd [term 1 LEADER]: Becoming Leader. State: Replica: 9eb2f1a7fb524990b88665f5acd19dbd, State: Running, Role: LEADER
I20260810 22:33:47.900328 19804 consensus_queue.cc:237] T d1c46c238ae240208afe8f3986bffe44 P 9eb2f1a7fb524990b88665f5acd19dbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3b49686ca5430db55bc282694fb9d4" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 43731 } } peers { permanent_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39973 } } peers { permanent_uuid: "029932492fa8492c80cc896ab4c881f6" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 40807 } }
I20260810 22:33:47.902568 19822 tablet_replica.cc:439] T d1c46c238ae240208afe8f3986bffe44 P 9eb2f1a7fb524990b88665f5acd19dbd: TxnStatusTablet state changed. Reason: New leader 9eb2f1a7fb524990b88665f5acd19dbd. Latest consensus state: current_term: 1 leader_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3b49686ca5430db55bc282694fb9d4" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 43731 } } peers { permanent_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39973 } } peers { permanent_uuid: "029932492fa8492c80cc896ab4c881f6" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 40807 } } }
I20260810 22:33:47.902701 19822 tablet_replica.cc:442] T d1c46c238ae240208afe8f3986bffe44 P 9eb2f1a7fb524990b88665f5acd19dbd: This TxnStatusTablet replica's current role is: LEADER
I20260810 22:33:47.902994 19824 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 22:33:47.903234 19824 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 22:33:47.904021 19228 catalog_manager.cc:5495] T d1c46c238ae240208afe8f3986bffe44 P 9eb2f1a7fb524990b88665f5acd19dbd reported cstate change: term changed from 0 to 1, leader changed from <none> to 9eb2f1a7fb524990b88665f5acd19dbd (127.16.233.66). New cstate: current_term: 1 leader_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3b49686ca5430db55bc282694fb9d4" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 43731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "029932492fa8492c80cc896ab4c881f6" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 40807 } health_report { overall_health: UNKNOWN } } }
I20260810 22:33:47.928203 19795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:47.924385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58406 (local address 127.16.233.66:39973)
0810 22:33:47.924512 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:47.924516 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:47.924533 (+ 17us) server_negotiation.cc:407] Connection header received
0810 22:33:47.924701 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:47.924704 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:47.924766 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:47.924920 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:33:47.925181 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:47.925699 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:47.926580 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:47.926805 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:47.926924 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:47.927267 (+ 343us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:33:47.927749 (+ 482us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:33:47.927887 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:47.927991 (+ 104us) server_negotiation.cc:299] Negotiation successful
0810 22:33:47.928048 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 22:33:47.928203 19816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:47.924231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.66:39973 (local address 127.0.0.1:58406)
0810 22:33:47.924382 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:47.924400 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:47.924523 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:47.924940 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:47.924943 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:47.924950 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 22:33:47.925069 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:47.925075 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:47.925819 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:47.925823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:47.926437 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:47.926459 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:47.926936 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:47.926940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:47.927031 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:47.927076 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 22:33:47.927860 (+ 784us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 22:33:47.927880 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 22:33:47.927961 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 22:33:47.928061 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260810 22:33:47.934288 19714 raft_consensus.cc:1261] T d1c46c238ae240208afe8f3986bffe44 P 029932492fa8492c80cc896ab4c881f6 [term 1 FOLLOWER]: Refusing update from remote peer 9eb2f1a7fb524990b88665f5acd19dbd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:33:47.934437 19404 raft_consensus.cc:1261] T d1c46c238ae240208afe8f3986bffe44 P 5d3b49686ca5430db55bc282694fb9d4 [term 1 FOLLOWER]: Refusing update from remote peer 9eb2f1a7fb524990b88665f5acd19dbd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:33:47.935051 19822 consensus_queue.cc:1035] T d1c46c238ae240208afe8f3986bffe44 P 9eb2f1a7fb524990b88665f5acd19dbd [LEADER]: Connected to new peer: Peer: permanent_uuid: "029932492fa8492c80cc896ab4c881f6" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:33:47.935585 19804 consensus_queue.cc:1035] T d1c46c238ae240208afe8f3986bffe44 P 9eb2f1a7fb524990b88665f5acd19dbd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d3b49686ca5430db55bc282694fb9d4" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 43731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:33:47.938743 19806 tablet_replica.cc:439] T d1c46c238ae240208afe8f3986bffe44 P 029932492fa8492c80cc896ab4c881f6: TxnStatusTablet state changed. Reason: New leader 9eb2f1a7fb524990b88665f5acd19dbd. Latest consensus state: current_term: 1 leader_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3b49686ca5430db55bc282694fb9d4" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 43731 } } peers { permanent_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39973 } } peers { permanent_uuid: "029932492fa8492c80cc896ab4c881f6" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 40807 } } }
I20260810 22:33:47.938876 19806 tablet_replica.cc:442] T d1c46c238ae240208afe8f3986bffe44 P 029932492fa8492c80cc896ab4c881f6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 22:33:47.940644 19805 tablet_replica.cc:439] T d1c46c238ae240208afe8f3986bffe44 P 5d3b49686ca5430db55bc282694fb9d4: TxnStatusTablet state changed. Reason: New leader 9eb2f1a7fb524990b88665f5acd19dbd. Latest consensus state: current_term: 1 leader_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3b49686ca5430db55bc282694fb9d4" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 43731 } } peers { permanent_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39973 } } peers { permanent_uuid: "029932492fa8492c80cc896ab4c881f6" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 40807 } } }
I20260810 22:33:47.940826 19805 tablet_replica.cc:442] T d1c46c238ae240208afe8f3986bffe44 P 5d3b49686ca5430db55bc282694fb9d4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 22:33:47.943466 19804 tablet_replica.cc:439] T d1c46c238ae240208afe8f3986bffe44 P 9eb2f1a7fb524990b88665f5acd19dbd: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3b49686ca5430db55bc282694fb9d4" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 43731 } } peers { permanent_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39973 } } peers { permanent_uuid: "029932492fa8492c80cc896ab4c881f6" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 40807 } } }
I20260810 22:33:47.943612 19804 tablet_replica.cc:442] T d1c46c238ae240208afe8f3986bffe44 P 9eb2f1a7fb524990b88665f5acd19dbd: This TxnStatusTablet replica's current role is: LEADER
I20260810 22:33:47.943493 19822 tablet_replica.cc:439] T d1c46c238ae240208afe8f3986bffe44 P 9eb2f1a7fb524990b88665f5acd19dbd: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3b49686ca5430db55bc282694fb9d4" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 43731 } } peers { permanent_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39973 } } peers { permanent_uuid: "029932492fa8492c80cc896ab4c881f6" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 40807 } } }
I20260810 22:33:47.943787 19822 tablet_replica.cc:442] T d1c46c238ae240208afe8f3986bffe44 P 9eb2f1a7fb524990b88665f5acd19dbd: This TxnStatusTablet replica's current role is: LEADER
I20260810 22:33:47.944339 19805 tablet_replica.cc:439] T d1c46c238ae240208afe8f3986bffe44 P 5d3b49686ca5430db55bc282694fb9d4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3b49686ca5430db55bc282694fb9d4" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 43731 } } peers { permanent_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39973 } } peers { permanent_uuid: "029932492fa8492c80cc896ab4c881f6" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 40807 } } }
I20260810 22:33:47.944481 19805 tablet_replica.cc:442] T d1c46c238ae240208afe8f3986bffe44 P 5d3b49686ca5430db55bc282694fb9d4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 22:33:47.944638 19806 tablet_replica.cc:439] T d1c46c238ae240208afe8f3986bffe44 P 029932492fa8492c80cc896ab4c881f6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3b49686ca5430db55bc282694fb9d4" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 43731 } } peers { permanent_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39973 } } peers { permanent_uuid: "029932492fa8492c80cc896ab4c881f6" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 40807 } } }
I20260810 22:33:47.944742 19806 tablet_replica.cc:442] T d1c46c238ae240208afe8f3986bffe44 P 029932492fa8492c80cc896ab4c881f6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 22:33:47.949203 19832 mvcc.cc:205] Tried to move back new op lower bound from 7317099429611229184 to 7317099429483331584. Current Snapshot: MvccSnapshot[applied={T|T < 7317099429611229184}]
I20260810 22:33:47.954093 19793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:47.950133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53398 (local address 127.16.233.67:40807)
0810 22:33:47.950261 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:47.950265 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:47.950515 (+ 250us) server_negotiation.cc:407] Connection header received
0810 22:33:47.950735 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:47.950738 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:47.950802 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:47.950917 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:33:47.951182 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:47.951694 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:47.952587 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:47.952881 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:47.953004 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:47.953281 (+ 277us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:33:47.953705 (+ 424us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:33:47.953838 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:47.953900 (+ 62us) server_negotiation.cc:299] Negotiation successful
0810 22:33:47.953954 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 22:33:47.965797 19795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:47.961786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:54459 (local address 127.16.233.66:39973)
0810 22:33:47.961923 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:47.961928 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:47.962114 (+ 186us) server_negotiation.cc:407] Connection header received
0810 22:33:47.962255 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:47.962258 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:47.962304 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:47.962369 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:33:47.962687 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:47.963191 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:47.964564 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:47.964776 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:47.964973 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:47.965370 (+ 397us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:33:47.965441 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:33:47.965497 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:47.965634 (+ 137us) server_negotiation.cc:299] Negotiation successful
0810 22:33:47.965682 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 22:33:47.965904 19843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:47.961627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.66:39973 (local address 127.16.233.67:54459)
0810 22:33:47.962007 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:47.962024 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:47.962147 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:47.962409 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:47.962412 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:47.962422 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 22:33:47.962558 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:47.962576 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:47.963308 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:47.963312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:47.964432 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:47.964443 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:47.965005 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:47.965010 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:47.965125 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:47.965186 (+ 61us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 22:33:47.965535 (+ 349us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 22:33:47.965542 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 22:33:47.965616 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 22:33:47.965703 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":138,"threads_started":1}
I20260810 22:33:47.975612 19404 raft_consensus.cc:1261] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 5d3b49686ca5430db55bc282694fb9d4 [term 1 FOLLOWER]: Refusing update from remote peer 029932492fa8492c80cc896ab4c881f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:33:47.975695 19563 raft_consensus.cc:1261] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 9eb2f1a7fb524990b88665f5acd19dbd [term 1 FOLLOWER]: Refusing update from remote peer 029932492fa8492c80cc896ab4c881f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:33:47.976224 19806 consensus_queue.cc:1035] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 029932492fa8492c80cc896ab4c881f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d3b49686ca5430db55bc282694fb9d4" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 43731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:33:47.976662 19808 consensus_queue.cc:1035] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 029932492fa8492c80cc896ab4c881f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:33:47.984912 19828 mvcc.cc:205] Tried to move back new op lower bound from 7317099429782065152 to 7317099428708208640. Current Snapshot: MvccSnapshot[applied={T|T < 7317099429782065152}]
I20260810 22:33:48.004613 19793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:48.000687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:52439 (local address 127.16.233.67:40807)
0810 22:33:48.000803 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:48.000835 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:48.000850 (+ 15us) server_negotiation.cc:407] Connection header received
0810 22:33:48.000955 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:48.000957 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:48.001007 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:48.001105 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:33:48.001373 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:48.001889 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:48.003071 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:48.003474 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:48.003602 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:48.004041 (+ 439us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:33:48.004137 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:33:48.004218 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:48.004409 (+ 191us) server_negotiation.cc:299] Negotiation successful
0810 22:33:48.004475 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 22:33:48.004660 19848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:48.000323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.67:40807 (local address 127.16.233.66:52439)
0810 22:33:48.000703 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:48.000721 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:48.000856 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:48.001114 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:48.001117 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:48.001125 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 22:33:48.001251 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:48.001257 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:48.002016 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:48.002022 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:48.002935 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:48.002945 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:48.003609 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:48.003611 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:48.003749 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:48.003819 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 22:33:48.004264 (+ 445us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 22:33:48.004272 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 22:33:48.004384 (+ 112us) client_negotiation.cc:240] Negotiation successful
0810 22:33:48.004475 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":153,"threads_started":1}
I20260810 22:33:48.030900 19224 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:33:48.031129 19224 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:33:48.033838 19224 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5652ecc5dafa4979ab57ef1506d00660: Sending DeleteTablet for 3 replicas of tablet 797b6cafc2c54861aeef1c1a0a3a1a80
I20260810 22:33:48.034652 19542 tablet_service.cc:1514] Processing DeleteTablet for tablet 797b6cafc2c54861aeef1c1a0a3a1a80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:33:48 UTC) from {username='kudu', principal='kudu/127.16.233.126@KRBTEST.COM'} at 127.0.0.1:58396
I20260810 22:33:48.034652 19692 tablet_service.cc:1514] Processing DeleteTablet for tablet 797b6cafc2c54861aeef1c1a0a3a1a80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:33:48 UTC) from {username='kudu', principal='kudu/127.16.233.126@KRBTEST.COM'} at 127.0.0.1:53392
I20260810 22:33:48.035388 19851 tablet_replica.cc:330] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 029932492fa8492c80cc896ab4c881f6: stopping tablet replica
I20260810 22:33:48.035662 19381 tablet_service.cc:1514] Processing DeleteTablet for tablet 797b6cafc2c54861aeef1c1a0a3a1a80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:33:48 UTC) from {username='kudu', principal='kudu/127.16.233.126@KRBTEST.COM'} at 127.0.0.1:36462
I20260810 22:33:48.035849 19851 raft_consensus.cc:2229] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 029932492fa8492c80cc896ab4c881f6 [term 1 LEADER]: Raft consensus shutting down.
I20260810 22:33:48.036351 19851 raft_consensus.cc:2258] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 029932492fa8492c80cc896ab4c881f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:33:48.036696 19853 tablet_replica.cc:330] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 5d3b49686ca5430db55bc282694fb9d4: stopping tablet replica
I20260810 22:33:48.037365 19224 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34292:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 22:33:48.037890 19224 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:33:48.038764 19851 ts_tablet_manager.cc:1905] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 029932492fa8492c80cc896ab4c881f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:33:48.039511 19853 raft_consensus.cc:2229] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 5d3b49686ca5430db55bc282694fb9d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:33:48.041873 19853 raft_consensus.cc:2258] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 5d3b49686ca5430db55bc282694fb9d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:33:48.042420 19852 tablet_replica.cc:330] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 9eb2f1a7fb524990b88665f5acd19dbd: stopping tablet replica
I20260810 22:33:48.044132 19381 tablet_service.cc:1467] Processing CreateTablet for tablet b0efb74c472d4ae2906fec4a46aa4922 (DEFAULT_TABLE table=test-table [id=0587f6fde89d41c5afd037295f38a2db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:33:48.044420 19381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0efb74c472d4ae2906fec4a46aa4922. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:33:48.044525 19853 ts_tablet_manager.cc:1905] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 5d3b49686ca5430db55bc282694fb9d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:33:48.044914 19692 tablet_service.cc:1467] Processing CreateTablet for tablet b0efb74c472d4ae2906fec4a46aa4922 (DEFAULT_TABLE table=test-table [id=0587f6fde89d41c5afd037295f38a2db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:33:48.045168 19692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0efb74c472d4ae2906fec4a46aa4922. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:33:48.047598 19852 raft_consensus.cc:2229] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 9eb2f1a7fb524990b88665f5acd19dbd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:33:48.047806 19852 raft_consensus.cc:2258] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 9eb2f1a7fb524990b88665f5acd19dbd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:33:48.047854 19799 tablet_bootstrap.cc:492] T b0efb74c472d4ae2906fec4a46aa4922 P 029932492fa8492c80cc896ab4c881f6: Bootstrap starting.
I20260810 22:33:48.049288 19542 tablet_service.cc:1467] Processing CreateTablet for tablet b0efb74c472d4ae2906fec4a46aa4922 (DEFAULT_TABLE table=test-table [id=0587f6fde89d41c5afd037295f38a2db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:33:48.049564 19542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0efb74c472d4ae2906fec4a46aa4922. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:33:48.050189 19852 ts_tablet_manager.cc:1905] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 9eb2f1a7fb524990b88665f5acd19dbd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:33:48.052089 19798 tablet_bootstrap.cc:492] T b0efb74c472d4ae2906fec4a46aa4922 P 9eb2f1a7fb524990b88665f5acd19dbd: Bootstrap starting.
I20260810 22:33:48.053092 19798 tablet_bootstrap.cc:654] T b0efb74c472d4ae2906fec4a46aa4922 P 9eb2f1a7fb524990b88665f5acd19dbd: Neither blocks nor log segments found. Creating new log.
I20260810 22:33:48.053267 19853 ts_tablet_manager.cc:1918] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 5d3b49686ca5430db55bc282694fb9d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 22:33:48.053385 19853 log.cc:1196] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 5d3b49686ca5430db55bc282694fb9d4: Deleting WAL directory at /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/ts-0/wal/wals/797b6cafc2c54861aeef1c1a0a3a1a80
I20260810 22:33:48.053834 19853 ts_tablet_manager.cc:1939] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 5d3b49686ca5430db55bc282694fb9d4: Deleting consensus metadata
I20260810 22:33:48.055284 19214 catalog_manager.cc:4841] TS 5d3b49686ca5430db55bc282694fb9d4 (127.16.233.65:43731): tablet 797b6cafc2c54861aeef1c1a0a3a1a80 (table test-table [id=8845afa1dada467689ab4949b3ff822a]) successfully deleted
I20260810 22:33:48.057134 19799 tablet_bootstrap.cc:654] T b0efb74c472d4ae2906fec4a46aa4922 P 029932492fa8492c80cc896ab4c881f6: Neither blocks nor log segments found. Creating new log.
I20260810 22:33:48.058765 19796 tablet_bootstrap.cc:492] T b0efb74c472d4ae2906fec4a46aa4922 P 5d3b49686ca5430db55bc282694fb9d4: Bootstrap starting.
I20260810 22:33:48.058951 19852 ts_tablet_manager.cc:1918] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 9eb2f1a7fb524990b88665f5acd19dbd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 22:33:48.059046 19852 log.cc:1196] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 9eb2f1a7fb524990b88665f5acd19dbd: Deleting WAL directory at /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/ts-1/wal/wals/797b6cafc2c54861aeef1c1a0a3a1a80
I20260810 22:33:48.059890 19796 tablet_bootstrap.cc:654] T b0efb74c472d4ae2906fec4a46aa4922 P 5d3b49686ca5430db55bc282694fb9d4: Neither blocks nor log segments found. Creating new log.
I20260810 22:33:48.060127 19798 tablet_bootstrap.cc:492] T b0efb74c472d4ae2906fec4a46aa4922 P 9eb2f1a7fb524990b88665f5acd19dbd: No bootstrap required, opened a new log
I20260810 22:33:48.060209 19798 ts_tablet_manager.cc:1397] T b0efb74c472d4ae2906fec4a46aa4922 P 9eb2f1a7fb524990b88665f5acd19dbd: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260810 22:33:48.061039 19798 raft_consensus.cc:348] T b0efb74c472d4ae2906fec4a46aa4922 P 9eb2f1a7fb524990b88665f5acd19dbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3b49686ca5430db55bc282694fb9d4" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 43731 } } peers { permanent_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39973 } } peers { permanent_uuid: "029932492fa8492c80cc896ab4c881f6" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 40807 } }
I20260810 22:33:48.061169 19798 raft_consensus.cc:374] T b0efb74c472d4ae2906fec4a46aa4922 P 9eb2f1a7fb524990b88665f5acd19dbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:33:48.061211 19798 raft_consensus.cc:729] T b0efb74c472d4ae2906fec4a46aa4922 P 9eb2f1a7fb524990b88665f5acd19dbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9eb2f1a7fb524990b88665f5acd19dbd, State: Initialized, Role: FOLLOWER
I20260810 22:33:48.062047 19851 ts_tablet_manager.cc:1918] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 029932492fa8492c80cc896ab4c881f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 22:33:48.062252 19851 log.cc:1196] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 029932492fa8492c80cc896ab4c881f6: Deleting WAL directory at /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/ts-2/wal/wals/797b6cafc2c54861aeef1c1a0a3a1a80
I20260810 22:33:48.062455 19798 consensus_queue.cc:260] T b0efb74c472d4ae2906fec4a46aa4922 P 9eb2f1a7fb524990b88665f5acd19dbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3b49686ca5430db55bc282694fb9d4" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 43731 } } peers { permanent_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39973 } } peers { permanent_uuid: "029932492fa8492c80cc896ab4c881f6" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 40807 } }
I20260810 22:33:48.062819 19798 ts_tablet_manager.cc:1428] T b0efb74c472d4ae2906fec4a46aa4922 P 9eb2f1a7fb524990b88665f5acd19dbd: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260810 22:33:48.063185 19852 ts_tablet_manager.cc:1939] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 9eb2f1a7fb524990b88665f5acd19dbd: Deleting consensus metadata
I20260810 22:33:48.064556 19213 catalog_manager.cc:4841] TS 9eb2f1a7fb524990b88665f5acd19dbd (127.16.233.66:39973): tablet 797b6cafc2c54861aeef1c1a0a3a1a80 (table test-table [id=8845afa1dada467689ab4949b3ff822a]) successfully deleted
I20260810 22:33:48.065116 19851 ts_tablet_manager.cc:1939] T 797b6cafc2c54861aeef1c1a0a3a1a80 P 029932492fa8492c80cc896ab4c881f6: Deleting consensus metadata
I20260810 22:33:48.065817 19799 tablet_bootstrap.cc:492] T b0efb74c472d4ae2906fec4a46aa4922 P 029932492fa8492c80cc896ab4c881f6: No bootstrap required, opened a new log
I20260810 22:33:48.065933 19799 ts_tablet_manager.cc:1397] T b0efb74c472d4ae2906fec4a46aa4922 P 029932492fa8492c80cc896ab4c881f6: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.002s
I20260810 22:33:48.066396 19799 raft_consensus.cc:348] T b0efb74c472d4ae2906fec4a46aa4922 P 029932492fa8492c80cc896ab4c881f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3b49686ca5430db55bc282694fb9d4" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 43731 } } peers { permanent_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39973 } } peers { permanent_uuid: "029932492fa8492c80cc896ab4c881f6" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 40807 } }
I20260810 22:33:48.066557 19799 raft_consensus.cc:374] T b0efb74c472d4ae2906fec4a46aa4922 P 029932492fa8492c80cc896ab4c881f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:33:48.066597 19799 raft_consensus.cc:729] T b0efb74c472d4ae2906fec4a46aa4922 P 029932492fa8492c80cc896ab4c881f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 029932492fa8492c80cc896ab4c881f6, State: Initialized, Role: FOLLOWER
I20260810 22:33:48.066606 19214 catalog_manager.cc:4841] TS 029932492fa8492c80cc896ab4c881f6 (127.16.233.67:40807): tablet 797b6cafc2c54861aeef1c1a0a3a1a80 (table test-table [id=8845afa1dada467689ab4949b3ff822a]) successfully deleted
I20260810 22:33:48.066726 19799 consensus_queue.cc:260] T b0efb74c472d4ae2906fec4a46aa4922 P 029932492fa8492c80cc896ab4c881f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3b49686ca5430db55bc282694fb9d4" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 43731 } } peers { permanent_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39973 } } peers { permanent_uuid: "029932492fa8492c80cc896ab4c881f6" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 40807 } }
I20260810 22:33:48.067040 19799 ts_tablet_manager.cc:1428] T b0efb74c472d4ae2906fec4a46aa4922 P 029932492fa8492c80cc896ab4c881f6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 22:33:48.068549 19796 tablet_bootstrap.cc:492] T b0efb74c472d4ae2906fec4a46aa4922 P 5d3b49686ca5430db55bc282694fb9d4: No bootstrap required, opened a new log
I20260810 22:33:48.068665 19796 ts_tablet_manager.cc:1397] T b0efb74c472d4ae2906fec4a46aa4922 P 5d3b49686ca5430db55bc282694fb9d4: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260810 22:33:48.069211 19796 raft_consensus.cc:348] T b0efb74c472d4ae2906fec4a46aa4922 P 5d3b49686ca5430db55bc282694fb9d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3b49686ca5430db55bc282694fb9d4" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 43731 } } peers { permanent_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39973 } } peers { permanent_uuid: "029932492fa8492c80cc896ab4c881f6" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 40807 } }
I20260810 22:33:48.069337 19796 raft_consensus.cc:374] T b0efb74c472d4ae2906fec4a46aa4922 P 5d3b49686ca5430db55bc282694fb9d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:33:48.069372 19796 raft_consensus.cc:729] T b0efb74c472d4ae2906fec4a46aa4922 P 5d3b49686ca5430db55bc282694fb9d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d3b49686ca5430db55bc282694fb9d4, State: Initialized, Role: FOLLOWER
I20260810 22:33:48.069525 19796 consensus_queue.cc:260] T b0efb74c472d4ae2906fec4a46aa4922 P 5d3b49686ca5430db55bc282694fb9d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3b49686ca5430db55bc282694fb9d4" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 43731 } } peers { permanent_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39973 } } peers { permanent_uuid: "029932492fa8492c80cc896ab4c881f6" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 40807 } }
I20260810 22:33:48.069840 19796 ts_tablet_manager.cc:1428] T b0efb74c472d4ae2906fec4a46aa4922 P 5d3b49686ca5430db55bc282694fb9d4: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 22:33:48.496110 19836 raft_consensus.cc:482] T b0efb74c472d4ae2906fec4a46aa4922 P 9eb2f1a7fb524990b88665f5acd19dbd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:33:48.496304 19836 raft_consensus.cc:504] T b0efb74c472d4ae2906fec4a46aa4922 P 9eb2f1a7fb524990b88665f5acd19dbd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3b49686ca5430db55bc282694fb9d4" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 43731 } } peers { permanent_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39973 } } peers { permanent_uuid: "029932492fa8492c80cc896ab4c881f6" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 40807 } }
I20260810 22:33:48.496904 19836 leader_election.cc:288] T b0efb74c472d4ae2906fec4a46aa4922 P 9eb2f1a7fb524990b88665f5acd19dbd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d3b49686ca5430db55bc282694fb9d4 (127.16.233.65:43731), 029932492fa8492c80cc896ab4c881f6 (127.16.233.67:40807)
I20260810 22:33:48.496910 19805 raft_consensus.cc:482] T b0efb74c472d4ae2906fec4a46aa4922 P 5d3b49686ca5430db55bc282694fb9d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:33:48.497058 19805 raft_consensus.cc:504] T b0efb74c472d4ae2906fec4a46aa4922 P 5d3b49686ca5430db55bc282694fb9d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3b49686ca5430db55bc282694fb9d4" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 43731 } } peers { permanent_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39973 } } peers { permanent_uuid: "029932492fa8492c80cc896ab4c881f6" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 40807 } }
I20260810 22:33:48.497483 19404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0efb74c472d4ae2906fec4a46aa4922" candidate_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d3b49686ca5430db55bc282694fb9d4" is_pre_election: true
I20260810 22:33:48.497532 19714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0efb74c472d4ae2906fec4a46aa4922" candidate_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "029932492fa8492c80cc896ab4c881f6" is_pre_election: true
I20260810 22:33:48.497632 19404 raft_consensus.cc:2454] T b0efb74c472d4ae2906fec4a46aa4922 P 5d3b49686ca5430db55bc282694fb9d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9eb2f1a7fb524990b88665f5acd19dbd in term 0.
I20260810 22:33:48.497664 19714 raft_consensus.cc:2454] T b0efb74c472d4ae2906fec4a46aa4922 P 029932492fa8492c80cc896ab4c881f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9eb2f1a7fb524990b88665f5acd19dbd in term 0.
I20260810 22:33:48.498059 19493 leader_election.cc:302] T b0efb74c472d4ae2906fec4a46aa4922 P 9eb2f1a7fb524990b88665f5acd19dbd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d3b49686ca5430db55bc282694fb9d4, 9eb2f1a7fb524990b88665f5acd19dbd; no voters:
I20260810 22:33:48.498385 19836 raft_consensus.cc:2781] T b0efb74c472d4ae2906fec4a46aa4922 P 9eb2f1a7fb524990b88665f5acd19dbd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:33:48.498425 19805 leader_election.cc:288] T b0efb74c472d4ae2906fec4a46aa4922 P 5d3b49686ca5430db55bc282694fb9d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9eb2f1a7fb524990b88665f5acd19dbd (127.16.233.66:39973), 029932492fa8492c80cc896ab4c881f6 (127.16.233.67:40807)
I20260810 22:33:48.498454 19836 raft_consensus.cc:482] T b0efb74c472d4ae2906fec4a46aa4922 P 9eb2f1a7fb524990b88665f5acd19dbd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:33:48.498504 19836 raft_consensus.cc:3037] T b0efb74c472d4ae2906fec4a46aa4922 P 9eb2f1a7fb524990b88665f5acd19dbd [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:33:48.499959 19836 raft_consensus.cc:504] T b0efb74c472d4ae2906fec4a46aa4922 P 9eb2f1a7fb524990b88665f5acd19dbd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3b49686ca5430db55bc282694fb9d4" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 43731 } } peers { permanent_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39973 } } peers { permanent_uuid: "029932492fa8492c80cc896ab4c881f6" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 40807 } }
I20260810 22:33:48.500349 19836 leader_election.cc:288] T b0efb74c472d4ae2906fec4a46aa4922 P 9eb2f1a7fb524990b88665f5acd19dbd [CANDIDATE]: Term 1 election: Requested vote from peers 5d3b49686ca5430db55bc282694fb9d4 (127.16.233.65:43731), 029932492fa8492c80cc896ab4c881f6 (127.16.233.67:40807)
I20260810 22:33:48.500757 19714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0efb74c472d4ae2906fec4a46aa4922" candidate_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "029932492fa8492c80cc896ab4c881f6"
I20260810 22:33:48.500773 19404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0efb74c472d4ae2906fec4a46aa4922" candidate_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d3b49686ca5430db55bc282694fb9d4"
I20260810 22:33:48.500921 19404 raft_consensus.cc:3037] T b0efb74c472d4ae2906fec4a46aa4922 P 5d3b49686ca5430db55bc282694fb9d4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:33:48.500923 19714 raft_consensus.cc:3037] T b0efb74c472d4ae2906fec4a46aa4922 P 029932492fa8492c80cc896ab4c881f6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:33:48.502137 19404 raft_consensus.cc:2454] T b0efb74c472d4ae2906fec4a46aa4922 P 5d3b49686ca5430db55bc282694fb9d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9eb2f1a7fb524990b88665f5acd19dbd in term 1.
I20260810 22:33:48.502210 19714 raft_consensus.cc:2454] T b0efb74c472d4ae2906fec4a46aa4922 P 029932492fa8492c80cc896ab4c881f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9eb2f1a7fb524990b88665f5acd19dbd in term 1.
I20260810 22:33:48.502564 19493 leader_election.cc:302] T b0efb74c472d4ae2906fec4a46aa4922 P 9eb2f1a7fb524990b88665f5acd19dbd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d3b49686ca5430db55bc282694fb9d4, 9eb2f1a7fb524990b88665f5acd19dbd; no voters:
I20260810 22:33:48.502909 19836 raft_consensus.cc:2781] T b0efb74c472d4ae2906fec4a46aa4922 P 9eb2f1a7fb524990b88665f5acd19dbd [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:33:48.503098 19836 raft_consensus.cc:686] T b0efb74c472d4ae2906fec4a46aa4922 P 9eb2f1a7fb524990b88665f5acd19dbd [term 1 LEADER]: Becoming Leader. State: Replica: 9eb2f1a7fb524990b88665f5acd19dbd, State: Running, Role: LEADER
I20260810 22:33:48.503259 19836 consensus_queue.cc:237] T b0efb74c472d4ae2906fec4a46aa4922 P 9eb2f1a7fb524990b88665f5acd19dbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3b49686ca5430db55bc282694fb9d4" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 43731 } } peers { permanent_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39973 } } peers { permanent_uuid: "029932492fa8492c80cc896ab4c881f6" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 40807 } }
I20260810 22:33:48.503708 19860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:48.498999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:33547 (local address 127.16.233.66:39973)
0810 22:33:48.499250 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:48.499257 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:48.499272 (+ 15us) server_negotiation.cc:407] Connection header received
0810 22:33:48.499333 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:48.499336 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:48.499383 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:48.499471 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:33:48.499793 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:48.500607 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:48.502218 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:48.502863 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:48.502984 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:48.503039 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:48.503466 (+ 427us) server_negotiation.cc:299] Negotiation successful
0810 22:33:48.503564 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":59,"threads_started":1}
I20260810 22:33:48.503715 19857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:48.498693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.66:39973 (local address 127.16.233.65:33547)
0810 22:33:48.499051 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:48.499069 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:48.499173 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:48.499513 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:48.499519 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:48.499549 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:33:48.499670 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:48.499679 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:48.500729 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:48.500732 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:48.502040 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:48.502054 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:48.503072 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:48.503079 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:48.503227 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:48.503255 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 22:33:48.503337 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 22:33:48.503532 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":1,"thread_start_us":89,"threads_started":1}
I20260810 22:33:48.503850 19793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:48.498767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:51741 (local address 127.16.233.67:40807)
0810 22:33:48.498910 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:48.498915 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:48.499549 (+ 634us) server_negotiation.cc:407] Connection header received
0810 22:33:48.499753 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:48.499756 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:48.499822 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:48.499897 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:33:48.500515 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:48.501230 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:48.502654 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:48.503207 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:48.503340 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:48.503396 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:48.503640 (+ 244us) server_negotiation.cc:299] Negotiation successful
0810 22:33:48.503714 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 22:33:48.504089 19859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:48.498693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.67:40807 (local address 127.16.233.65:51741)
0810 22:33:48.499463 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:48.499477 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:48.499630 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:48.499933 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:48.499937 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:48.499958 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:33:48.500344 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:48.500356 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:48.501374 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:48.501377 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:48.502512 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:48.502522 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:48.503386 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:48.503391 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:48.503499 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:48.503521 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 22:33:48.503855 (+ 334us) client_negotiation.cc:240] Negotiation successful
0810 22:33:48.503941 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"mutex_wait_us":81,"spinlock_wait_cycles":768,"thread_start_us":66,"threads_started":1}
I20260810 22:33:48.504575 19714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0efb74c472d4ae2906fec4a46aa4922" candidate_uuid: "5d3b49686ca5430db55bc282694fb9d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "029932492fa8492c80cc896ab4c881f6" is_pre_election: true
I20260810 22:33:48.504608 19563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0efb74c472d4ae2906fec4a46aa4922" candidate_uuid: "5d3b49686ca5430db55bc282694fb9d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" is_pre_election: true
I20260810 22:33:48.504774 19714 raft_consensus.cc:2379] T b0efb74c472d4ae2906fec4a46aa4922 P 029932492fa8492c80cc896ab4c881f6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5d3b49686ca5430db55bc282694fb9d4 in current term 1: Already voted for candidate 9eb2f1a7fb524990b88665f5acd19dbd in this term.
I20260810 22:33:48.505225 19326 leader_election.cc:302] T b0efb74c472d4ae2906fec4a46aa4922 P 5d3b49686ca5430db55bc282694fb9d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d3b49686ca5430db55bc282694fb9d4; no voters: 029932492fa8492c80cc896ab4c881f6, 9eb2f1a7fb524990b88665f5acd19dbd
I20260810 22:33:48.505565 19805 raft_consensus.cc:2726] T b0efb74c472d4ae2906fec4a46aa4922 P 5d3b49686ca5430db55bc282694fb9d4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 22:33:48.505887 19224 catalog_manager.cc:5495] T b0efb74c472d4ae2906fec4a46aa4922 P 9eb2f1a7fb524990b88665f5acd19dbd reported cstate change: term changed from 0 to 1, leader changed from <none> to 9eb2f1a7fb524990b88665f5acd19dbd (127.16.233.66). New cstate: current_term: 1 leader_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3b49686ca5430db55bc282694fb9d4" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 43731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "029932492fa8492c80cc896ab4c881f6" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 40807 } health_report { overall_health: UNKNOWN } } }
I20260810 22:33:48.623181 19860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:48.620076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58418 (local address 127.16.233.66:39973)
0810 22:33:48.620208 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:48.620211 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:48.620259 (+ 48us) server_negotiation.cc:407] Connection header received
0810 22:33:48.620361 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:48.620363 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:48.620406 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:48.620481 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:33:48.620691 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:48.621353 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:48.622192 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:48.622426 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:48.622511 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:48.622739 (+ 228us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:33:48.622859 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:33:48.622919 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:48.623016 (+ 97us) server_negotiation.cc:299] Negotiation successful
0810 22:33:48.623065 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 22:33:48.633167 19860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:48.629793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:39469 (local address 127.16.233.66:39973)
0810 22:33:48.629955 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:48.629961 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:48.630123 (+ 162us) server_negotiation.cc:407] Connection header received
0810 22:33:48.630309 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:48.630312 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:48.630357 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:48.630433 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:33:48.630717 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:48.631230 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:48.632094 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:48.632306 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:48.632419 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:48.632650 (+ 231us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:33:48.632731 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:33:48.632809 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:48.632921 (+ 112us) server_negotiation.cc:299] Negotiation successful
0810 22:33:48.632999 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":33}
I20260810 22:33:48.633181 19874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:48.629669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.66:39973 (local address 127.16.233.66:39469)
0810 22:33:48.630017 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:48.630030 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:48.630167 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:48.630453 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:48.630455 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:48.630462 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 22:33:48.630607 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:48.630614 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:48.631352 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:48.631355 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:48.631970 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:48.631978 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:48.632421 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:48.632424 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:48.632504 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:48.632535 (+ 31us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 22:33:48.632843 (+ 308us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 22:33:48.632851 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 22:33:48.632912 (+ 61us) client_negotiation.cc:240] Negotiation successful
0810 22:33:48.633020 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"mutex_wait_us":34,"thread_start_us":119,"threads_started":1}
I20260810 22:33:48.639717 19404 raft_consensus.cc:1261] T b0efb74c472d4ae2906fec4a46aa4922 P 5d3b49686ca5430db55bc282694fb9d4 [term 1 FOLLOWER]: Refusing update from remote peer 9eb2f1a7fb524990b88665f5acd19dbd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:33:48.639779 19714 raft_consensus.cc:1261] T b0efb74c472d4ae2906fec4a46aa4922 P 029932492fa8492c80cc896ab4c881f6 [term 1 FOLLOWER]: Refusing update from remote peer 9eb2f1a7fb524990b88665f5acd19dbd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:33:48.640257 19836 consensus_queue.cc:1035] T b0efb74c472d4ae2906fec4a46aa4922 P 9eb2f1a7fb524990b88665f5acd19dbd [LEADER]: Connected to new peer: Peer: permanent_uuid: "029932492fa8492c80cc896ab4c881f6" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:33:48.640429 19866 consensus_queue.cc:1035] T b0efb74c472d4ae2906fec4a46aa4922 P 9eb2f1a7fb524990b88665f5acd19dbd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d3b49686ca5430db55bc282694fb9d4" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 43731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 22:33:48.655329 19863 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260810 22:33:50.235504 19227 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:33:50.235682 19227 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:33:50.238909 19227 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5652ecc5dafa4979ab57ef1506d00660: Sending DeleteTablet for 3 replicas of tablet b0efb74c472d4ae2906fec4a46aa4922
I20260810 22:33:50.239717 19542 tablet_service.cc:1514] Processing DeleteTablet for tablet b0efb74c472d4ae2906fec4a46aa4922 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:33:50 UTC) from {username='kudu', principal='kudu/127.16.233.126@KRBTEST.COM'} at 127.0.0.1:58396
I20260810 22:33:50.239877 19692 tablet_service.cc:1514] Processing DeleteTablet for tablet b0efb74c472d4ae2906fec4a46aa4922 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:33:50 UTC) from {username='kudu', principal='kudu/127.16.233.126@KRBTEST.COM'} at 127.0.0.1:53392
I20260810 22:33:50.239934 19381 tablet_service.cc:1514] Processing DeleteTablet for tablet b0efb74c472d4ae2906fec4a46aa4922 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:33:50 UTC) from {username='kudu', principal='kudu/127.16.233.126@KRBTEST.COM'} at 127.0.0.1:36462
I20260810 22:33:50.240190 19905 tablet_replica.cc:330] T b0efb74c472d4ae2906fec4a46aa4922 P 9eb2f1a7fb524990b88665f5acd19dbd: stopping tablet replica
I20260810 22:33:50.240356 19905 raft_consensus.cc:2229] T b0efb74c472d4ae2906fec4a46aa4922 P 9eb2f1a7fb524990b88665f5acd19dbd [term 1 LEADER]: Raft consensus shutting down.
I20260810 22:33:50.240728 19905 raft_consensus.cc:2258] T b0efb74c472d4ae2906fec4a46aa4922 P 9eb2f1a7fb524990b88665f5acd19dbd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:33:50.241576 19905 ts_tablet_manager.cc:1905] T b0efb74c472d4ae2906fec4a46aa4922 P 9eb2f1a7fb524990b88665f5acd19dbd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:33:50.243113 19906 tablet_replica.cc:330] T b0efb74c472d4ae2906fec4a46aa4922 P 029932492fa8492c80cc896ab4c881f6: stopping tablet replica
I20260810 22:33:50.243314 19906 raft_consensus.cc:2229] T b0efb74c472d4ae2906fec4a46aa4922 P 029932492fa8492c80cc896ab4c881f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:33:50.243487 19906 raft_consensus.cc:2258] T b0efb74c472d4ae2906fec4a46aa4922 P 029932492fa8492c80cc896ab4c881f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:33:50.243727 19909 tablet_replica.cc:330] T b0efb74c472d4ae2906fec4a46aa4922 P 5d3b49686ca5430db55bc282694fb9d4: stopping tablet replica
I20260810 22:33:50.243933 19909 raft_consensus.cc:2229] T b0efb74c472d4ae2906fec4a46aa4922 P 5d3b49686ca5430db55bc282694fb9d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:33:50.244120 19909 raft_consensus.cc:2258] T b0efb74c472d4ae2906fec4a46aa4922 P 5d3b49686ca5430db55bc282694fb9d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:33:50.244946 19906 ts_tablet_manager.cc:1905] T b0efb74c472d4ae2906fec4a46aa4922 P 029932492fa8492c80cc896ab4c881f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:33:50.244954 19909 ts_tablet_manager.cc:1905] T b0efb74c472d4ae2906fec4a46aa4922 P 5d3b49686ca5430db55bc282694fb9d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:33:50.245368 19905 ts_tablet_manager.cc:1918] T b0efb74c472d4ae2906fec4a46aa4922 P 9eb2f1a7fb524990b88665f5acd19dbd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 22:33:50.245465 19905 log.cc:1196] T b0efb74c472d4ae2906fec4a46aa4922 P 9eb2f1a7fb524990b88665f5acd19dbd: Deleting WAL directory at /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/ts-1/wal/wals/b0efb74c472d4ae2906fec4a46aa4922
I20260810 22:33:50.245790 19905 ts_tablet_manager.cc:1939] T b0efb74c472d4ae2906fec4a46aa4922 P 9eb2f1a7fb524990b88665f5acd19dbd: Deleting consensus metadata
I20260810 22:33:50.246790 19213 catalog_manager.cc:4841] TS 9eb2f1a7fb524990b88665f5acd19dbd (127.16.233.66:39973): tablet b0efb74c472d4ae2906fec4a46aa4922 (table test-table [id=0587f6fde89d41c5afd037295f38a2db]) successfully deleted
I20260810 22:33:50.248023 19909 ts_tablet_manager.cc:1918] T b0efb74c472d4ae2906fec4a46aa4922 P 5d3b49686ca5430db55bc282694fb9d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 22:33:50.248126 19909 log.cc:1196] T b0efb74c472d4ae2906fec4a46aa4922 P 5d3b49686ca5430db55bc282694fb9d4: Deleting WAL directory at /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/ts-0/wal/wals/b0efb74c472d4ae2906fec4a46aa4922
I20260810 22:33:50.248132 19906 ts_tablet_manager.cc:1918] T b0efb74c472d4ae2906fec4a46aa4922 P 029932492fa8492c80cc896ab4c881f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 22:33:50.248262 19906 log.cc:1196] T b0efb74c472d4ae2906fec4a46aa4922 P 029932492fa8492c80cc896ab4c881f6: Deleting WAL directory at /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/ts-2/wal/wals/b0efb74c472d4ae2906fec4a46aa4922
I20260810 22:33:50.248550 19909 ts_tablet_manager.cc:1939] T b0efb74c472d4ae2906fec4a46aa4922 P 5d3b49686ca5430db55bc282694fb9d4: Deleting consensus metadata
I20260810 22:33:50.248587 19906 ts_tablet_manager.cc:1939] T b0efb74c472d4ae2906fec4a46aa4922 P 029932492fa8492c80cc896ab4c881f6: Deleting consensus metadata
I20260810 22:33:50.249491 19214 catalog_manager.cc:4841] TS 5d3b49686ca5430db55bc282694fb9d4 (127.16.233.65:43731): tablet b0efb74c472d4ae2906fec4a46aa4922 (table test-table [id=0587f6fde89d41c5afd037295f38a2db]) successfully deleted
I20260810 22:33:50.249671 19214 catalog_manager.cc:4841] TS 029932492fa8492c80cc896ab4c881f6 (127.16.233.67:40807): tablet b0efb74c472d4ae2906fec4a46aa4922 (table test-table [id=0587f6fde89d41c5afd037295f38a2db]) successfully deleted
Aug 10 22:33:50 dist-test-slave-77v9 krb5kdc[19178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401230, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 22:33:50 dist-test-slave-77v9 krb5kdc[19178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401230, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:33:50.275234 19918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:50.264726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53126 (local address 127.16.233.126:43475)
0810 22:33:50.265150 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:50.265158 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:50.265185 (+ 27us) server_negotiation.cc:407] Connection header received
0810 22:33:50.265303 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:50.265305 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:50.265373 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:50.265460 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:50.266330 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:50.266972 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:50.267935 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:50.268180 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:50.268277 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:50.271294 (+ 3017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:50.271298 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:50.271300 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:50.271334 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:50.273228 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:50.273734 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:50.273737 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:50.273738 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:50.273846 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:50.274172 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:50.274174 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:50.274176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:50.274408 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:50.274525 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:50.274851 (+ 326us) server_negotiation.cc:299] Negotiation successful
0810 22:33:50.274987 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":205,"threads_started":1}
I20260810 22:33:50.278427 19227 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53126:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 22:33:50.278941 19227 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:33:50.284554 19542 tablet_service.cc:1467] Processing CreateTablet for tablet 4e049c434d8d46f4a1b3d0ff08a37268 (DEFAULT_TABLE table=test-table [id=161bbe42111c4417859ebe5d13670645]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:33:50.284626 19692 tablet_service.cc:1467] Processing CreateTablet for tablet 4e049c434d8d46f4a1b3d0ff08a37268 (DEFAULT_TABLE table=test-table [id=161bbe42111c4417859ebe5d13670645]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:33:50.284976 19542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e049c434d8d46f4a1b3d0ff08a37268. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:33:50.285018 19692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e049c434d8d46f4a1b3d0ff08a37268. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:33:50.287338 19919 tablet_bootstrap.cc:492] T 4e049c434d8d46f4a1b3d0ff08a37268 P 9eb2f1a7fb524990b88665f5acd19dbd: Bootstrap starting.
I20260810 22:33:50.287581 19381 tablet_service.cc:1467] Processing CreateTablet for tablet 4e049c434d8d46f4a1b3d0ff08a37268 (DEFAULT_TABLE table=test-table [id=161bbe42111c4417859ebe5d13670645]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:33:50.287891 19920 tablet_bootstrap.cc:492] T 4e049c434d8d46f4a1b3d0ff08a37268 P 029932492fa8492c80cc896ab4c881f6: Bootstrap starting.
I20260810 22:33:50.287895 19381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e049c434d8d46f4a1b3d0ff08a37268. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:33:50.288420 19919 tablet_bootstrap.cc:654] T 4e049c434d8d46f4a1b3d0ff08a37268 P 9eb2f1a7fb524990b88665f5acd19dbd: Neither blocks nor log segments found. Creating new log.
I20260810 22:33:50.288996 19920 tablet_bootstrap.cc:654] T 4e049c434d8d46f4a1b3d0ff08a37268 P 029932492fa8492c80cc896ab4c881f6: Neither blocks nor log segments found. Creating new log.
I20260810 22:33:50.289896 19919 tablet_bootstrap.cc:492] T 4e049c434d8d46f4a1b3d0ff08a37268 P 9eb2f1a7fb524990b88665f5acd19dbd: No bootstrap required, opened a new log
I20260810 22:33:50.290011 19919 ts_tablet_manager.cc:1397] T 4e049c434d8d46f4a1b3d0ff08a37268 P 9eb2f1a7fb524990b88665f5acd19dbd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 22:33:50.290256 19920 tablet_bootstrap.cc:492] T 4e049c434d8d46f4a1b3d0ff08a37268 P 029932492fa8492c80cc896ab4c881f6: No bootstrap required, opened a new log
I20260810 22:33:50.290344 19920 ts_tablet_manager.cc:1397] T 4e049c434d8d46f4a1b3d0ff08a37268 P 029932492fa8492c80cc896ab4c881f6: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260810 22:33:50.290494 19919 raft_consensus.cc:348] T 4e049c434d8d46f4a1b3d0ff08a37268 P 9eb2f1a7fb524990b88665f5acd19dbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3b49686ca5430db55bc282694fb9d4" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 43731 } } peers { permanent_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39973 } } peers { permanent_uuid: "029932492fa8492c80cc896ab4c881f6" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 40807 } }
I20260810 22:33:50.290614 19919 raft_consensus.cc:374] T 4e049c434d8d46f4a1b3d0ff08a37268 P 9eb2f1a7fb524990b88665f5acd19dbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:33:50.290647 19919 raft_consensus.cc:729] T 4e049c434d8d46f4a1b3d0ff08a37268 P 9eb2f1a7fb524990b88665f5acd19dbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9eb2f1a7fb524990b88665f5acd19dbd, State: Initialized, Role: FOLLOWER
I20260810 22:33:50.290740 19920 raft_consensus.cc:348] T 4e049c434d8d46f4a1b3d0ff08a37268 P 029932492fa8492c80cc896ab4c881f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3b49686ca5430db55bc282694fb9d4" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 43731 } } peers { permanent_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39973 } } peers { permanent_uuid: "029932492fa8492c80cc896ab4c881f6" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 40807 } }
I20260810 22:33:50.290784 19919 consensus_queue.cc:260] T 4e049c434d8d46f4a1b3d0ff08a37268 P 9eb2f1a7fb524990b88665f5acd19dbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3b49686ca5430db55bc282694fb9d4" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 43731 } } peers { permanent_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39973 } } peers { permanent_uuid: "029932492fa8492c80cc896ab4c881f6" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 40807 } }
I20260810 22:33:50.290841 19920 raft_consensus.cc:374] T 4e049c434d8d46f4a1b3d0ff08a37268 P 029932492fa8492c80cc896ab4c881f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:33:50.290885 19920 raft_consensus.cc:729] T 4e049c434d8d46f4a1b3d0ff08a37268 P 029932492fa8492c80cc896ab4c881f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 029932492fa8492c80cc896ab4c881f6, State: Initialized, Role: FOLLOWER
I20260810 22:33:50.291091 19919 ts_tablet_manager.cc:1428] T 4e049c434d8d46f4a1b3d0ff08a37268 P 9eb2f1a7fb524990b88665f5acd19dbd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 22:33:50.291061 19920 consensus_queue.cc:260] T 4e049c434d8d46f4a1b3d0ff08a37268 P 029932492fa8492c80cc896ab4c881f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3b49686ca5430db55bc282694fb9d4" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 43731 } } peers { permanent_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39973 } } peers { permanent_uuid: "029932492fa8492c80cc896ab4c881f6" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 40807 } }
I20260810 22:33:50.291486 19920 ts_tablet_manager.cc:1428] T 4e049c434d8d46f4a1b3d0ff08a37268 P 029932492fa8492c80cc896ab4c881f6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 22:33:50.291798 19923 tablet_bootstrap.cc:492] T 4e049c434d8d46f4a1b3d0ff08a37268 P 5d3b49686ca5430db55bc282694fb9d4: Bootstrap starting.
I20260810 22:33:50.293463 19923 tablet_bootstrap.cc:654] T 4e049c434d8d46f4a1b3d0ff08a37268 P 5d3b49686ca5430db55bc282694fb9d4: Neither blocks nor log segments found. Creating new log.
I20260810 22:33:50.295042 19923 tablet_bootstrap.cc:492] T 4e049c434d8d46f4a1b3d0ff08a37268 P 5d3b49686ca5430db55bc282694fb9d4: No bootstrap required, opened a new log
I20260810 22:33:50.295147 19923 ts_tablet_manager.cc:1397] T 4e049c434d8d46f4a1b3d0ff08a37268 P 5d3b49686ca5430db55bc282694fb9d4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 22:33:50.295608 19923 raft_consensus.cc:348] T 4e049c434d8d46f4a1b3d0ff08a37268 P 5d3b49686ca5430db55bc282694fb9d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3b49686ca5430db55bc282694fb9d4" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 43731 } } peers { permanent_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39973 } } peers { permanent_uuid: "029932492fa8492c80cc896ab4c881f6" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 40807 } }
I20260810 22:33:50.295714 19923 raft_consensus.cc:374] T 4e049c434d8d46f4a1b3d0ff08a37268 P 5d3b49686ca5430db55bc282694fb9d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:33:50.295737 19923 raft_consensus.cc:729] T 4e049c434d8d46f4a1b3d0ff08a37268 P 5d3b49686ca5430db55bc282694fb9d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d3b49686ca5430db55bc282694fb9d4, State: Initialized, Role: FOLLOWER
I20260810 22:33:50.295892 19923 consensus_queue.cc:260] T 4e049c434d8d46f4a1b3d0ff08a37268 P 5d3b49686ca5430db55bc282694fb9d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3b49686ca5430db55bc282694fb9d4" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 43731 } } peers { permanent_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39973 } } peers { permanent_uuid: "029932492fa8492c80cc896ab4c881f6" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 40807 } }
I20260810 22:33:50.296298 19923 ts_tablet_manager.cc:1428] T 4e049c434d8d46f4a1b3d0ff08a37268 P 5d3b49686ca5430db55bc282694fb9d4: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260810 22:33:50.313606 19891 raft_consensus.cc:482] T 4e049c434d8d46f4a1b3d0ff08a37268 P 9eb2f1a7fb524990b88665f5acd19dbd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:33:50.313761 19891 raft_consensus.cc:504] T 4e049c434d8d46f4a1b3d0ff08a37268 P 9eb2f1a7fb524990b88665f5acd19dbd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3b49686ca5430db55bc282694fb9d4" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 43731 } } peers { permanent_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39973 } } peers { permanent_uuid: "029932492fa8492c80cc896ab4c881f6" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 40807 } }
I20260810 22:33:50.314208 19891 leader_election.cc:288] T 4e049c434d8d46f4a1b3d0ff08a37268 P 9eb2f1a7fb524990b88665f5acd19dbd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d3b49686ca5430db55bc282694fb9d4 (127.16.233.65:43731), 029932492fa8492c80cc896ab4c881f6 (127.16.233.67:40807)
I20260810 22:33:50.314762 19404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e049c434d8d46f4a1b3d0ff08a37268" candidate_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d3b49686ca5430db55bc282694fb9d4" is_pre_election: true
I20260810 22:33:50.314956 19404 raft_consensus.cc:2454] T 4e049c434d8d46f4a1b3d0ff08a37268 P 5d3b49686ca5430db55bc282694fb9d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9eb2f1a7fb524990b88665f5acd19dbd in term 0.
I20260810 22:33:50.314993 19715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e049c434d8d46f4a1b3d0ff08a37268" candidate_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "029932492fa8492c80cc896ab4c881f6" is_pre_election: true
I20260810 22:33:50.315199 19715 raft_consensus.cc:2454] T 4e049c434d8d46f4a1b3d0ff08a37268 P 029932492fa8492c80cc896ab4c881f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9eb2f1a7fb524990b88665f5acd19dbd in term 0.
I20260810 22:33:50.315390 19493 leader_election.cc:302] T 4e049c434d8d46f4a1b3d0ff08a37268 P 9eb2f1a7fb524990b88665f5acd19dbd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d3b49686ca5430db55bc282694fb9d4, 9eb2f1a7fb524990b88665f5acd19dbd; no voters:
I20260810 22:33:50.315587 19891 raft_consensus.cc:2781] T 4e049c434d8d46f4a1b3d0ff08a37268 P 9eb2f1a7fb524990b88665f5acd19dbd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:33:50.315649 19891 raft_consensus.cc:482] T 4e049c434d8d46f4a1b3d0ff08a37268 P 9eb2f1a7fb524990b88665f5acd19dbd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:33:50.315676 19891 raft_consensus.cc:3037] T 4e049c434d8d46f4a1b3d0ff08a37268 P 9eb2f1a7fb524990b88665f5acd19dbd [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:33:50.317020 19891 raft_consensus.cc:504] T 4e049c434d8d46f4a1b3d0ff08a37268 P 9eb2f1a7fb524990b88665f5acd19dbd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3b49686ca5430db55bc282694fb9d4" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 43731 } } peers { permanent_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39973 } } peers { permanent_uuid: "029932492fa8492c80cc896ab4c881f6" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 40807 } }
I20260810 22:33:50.317442 19891 leader_election.cc:288] T 4e049c434d8d46f4a1b3d0ff08a37268 P 9eb2f1a7fb524990b88665f5acd19dbd [CANDIDATE]: Term 1 election: Requested vote from peers 5d3b49686ca5430db55bc282694fb9d4 (127.16.233.65:43731), 029932492fa8492c80cc896ab4c881f6 (127.16.233.67:40807)
I20260810 22:33:50.317883 19404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e049c434d8d46f4a1b3d0ff08a37268" candidate_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d3b49686ca5430db55bc282694fb9d4"
I20260810 22:33:50.317952 19715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e049c434d8d46f4a1b3d0ff08a37268" candidate_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "029932492fa8492c80cc896ab4c881f6"
I20260810 22:33:50.318040 19404 raft_consensus.cc:3037] T 4e049c434d8d46f4a1b3d0ff08a37268 P 5d3b49686ca5430db55bc282694fb9d4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:33:50.318118 19715 raft_consensus.cc:3037] T 4e049c434d8d46f4a1b3d0ff08a37268 P 029932492fa8492c80cc896ab4c881f6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:33:50.319017 19404 raft_consensus.cc:2454] T 4e049c434d8d46f4a1b3d0ff08a37268 P 5d3b49686ca5430db55bc282694fb9d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9eb2f1a7fb524990b88665f5acd19dbd in term 1.
I20260810 22:33:50.319268 19715 raft_consensus.cc:2454] T 4e049c434d8d46f4a1b3d0ff08a37268 P 029932492fa8492c80cc896ab4c881f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9eb2f1a7fb524990b88665f5acd19dbd in term 1.
I20260810 22:33:50.319345 19493 leader_election.cc:302] T 4e049c434d8d46f4a1b3d0ff08a37268 P 9eb2f1a7fb524990b88665f5acd19dbd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d3b49686ca5430db55bc282694fb9d4, 9eb2f1a7fb524990b88665f5acd19dbd; no voters:
I20260810 22:33:50.319541 19891 raft_consensus.cc:2781] T 4e049c434d8d46f4a1b3d0ff08a37268 P 9eb2f1a7fb524990b88665f5acd19dbd [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:33:50.319716 19891 raft_consensus.cc:686] T 4e049c434d8d46f4a1b3d0ff08a37268 P 9eb2f1a7fb524990b88665f5acd19dbd [term 1 LEADER]: Becoming Leader. State: Replica: 9eb2f1a7fb524990b88665f5acd19dbd, State: Running, Role: LEADER
I20260810 22:33:50.319898 19891 consensus_queue.cc:237] T 4e049c434d8d46f4a1b3d0ff08a37268 P 9eb2f1a7fb524990b88665f5acd19dbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3b49686ca5430db55bc282694fb9d4" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 43731 } } peers { permanent_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39973 } } peers { permanent_uuid: "029932492fa8492c80cc896ab4c881f6" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 40807 } }
I20260810 22:33:50.321985 19227 catalog_manager.cc:5495] T 4e049c434d8d46f4a1b3d0ff08a37268 P 9eb2f1a7fb524990b88665f5acd19dbd reported cstate change: term changed from 0 to 1, leader changed from <none> to 9eb2f1a7fb524990b88665f5acd19dbd (127.16.233.66). New cstate: current_term: 1 leader_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3b49686ca5430db55bc282694fb9d4" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 43731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "029932492fa8492c80cc896ab4c881f6" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 40807 } health_report { overall_health: UNKNOWN } } }
I20260810 22:33:50.339893 19929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:50.336401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57362 (local address 127.16.233.66:39973)
0810 22:33:50.336757 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:50.336769 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:50.336795 (+ 26us) server_negotiation.cc:407] Connection header received
0810 22:33:50.336874 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:50.336877 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:50.336937 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:50.337033 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:33:50.337313 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:50.337857 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:50.338728 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:50.338980 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:50.339093 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:50.339373 (+ 280us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:33:50.339459 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:33:50.339551 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:50.339673 (+ 122us) server_negotiation.cc:299] Negotiation successful
0810 22:33:50.339729 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":111,"threads_started":1}
I20260810 22:33:50.347582 19404 raft_consensus.cc:1261] T 4e049c434d8d46f4a1b3d0ff08a37268 P 5d3b49686ca5430db55bc282694fb9d4 [term 1 FOLLOWER]: Refusing update from remote peer 9eb2f1a7fb524990b88665f5acd19dbd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:33:50.347607 19715 raft_consensus.cc:1261] T 4e049c434d8d46f4a1b3d0ff08a37268 P 029932492fa8492c80cc896ab4c881f6 [term 1 FOLLOWER]: Refusing update from remote peer 9eb2f1a7fb524990b88665f5acd19dbd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:33:50.348044 19883 consensus_queue.cc:1035] T 4e049c434d8d46f4a1b3d0ff08a37268 P 9eb2f1a7fb524990b88665f5acd19dbd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d3b49686ca5430db55bc282694fb9d4" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 43731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:33:50.348255 19836 consensus_queue.cc:1035] T 4e049c434d8d46f4a1b3d0ff08a37268 P 9eb2f1a7fb524990b88665f5acd19dbd [LEADER]: Connected to new peer: Peer: permanent_uuid: "029932492fa8492c80cc896ab4c881f6" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:33:50.352990 19886 mvcc.cc:205] Tried to move back new op lower bound from 7317099439500075008 to 7317099439391789056. Current Snapshot: MvccSnapshot[applied={T|T < 7317099439500075008}]
I20260810 22:33:50.385911 19227 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53126:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:33:50.386096 19227 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53126:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:33:50.388670 19227 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5652ecc5dafa4979ab57ef1506d00660: Sending DeleteTablet for 3 replicas of tablet 4e049c434d8d46f4a1b3d0ff08a37268
I20260810 22:33:50.389506 19542 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e049c434d8d46f4a1b3d0ff08a37268 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:33:50 UTC) from {username='kudu', principal='kudu/127.16.233.126@KRBTEST.COM'} at 127.0.0.1:58396
I20260810 22:33:50.389513 19381 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e049c434d8d46f4a1b3d0ff08a37268 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:33:50 UTC) from {username='kudu', principal='kudu/127.16.233.126@KRBTEST.COM'} at 127.0.0.1:36462
I20260810 22:33:50.389571 19692 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e049c434d8d46f4a1b3d0ff08a37268 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:33:50 UTC) from {username='kudu', principal='kudu/127.16.233.126@KRBTEST.COM'} at 127.0.0.1:53392
I20260810 22:33:50.389741 19905 tablet_replica.cc:330] T 4e049c434d8d46f4a1b3d0ff08a37268 P 9eb2f1a7fb524990b88665f5acd19dbd: stopping tablet replica
I20260810 22:33:50.389786 19909 tablet_replica.cc:330] T 4e049c434d8d46f4a1b3d0ff08a37268 P 5d3b49686ca5430db55bc282694fb9d4: stopping tablet replica
I20260810 22:33:50.389876 19905 raft_consensus.cc:2229] T 4e049c434d8d46f4a1b3d0ff08a37268 P 9eb2f1a7fb524990b88665f5acd19dbd [term 1 LEADER]: Raft consensus shutting down.
I20260810 22:33:50.389892 19909 raft_consensus.cc:2229] T 4e049c434d8d46f4a1b3d0ff08a37268 P 5d3b49686ca5430db55bc282694fb9d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:33:50.389992 19909 raft_consensus.cc:2258] T 4e049c434d8d46f4a1b3d0ff08a37268 P 5d3b49686ca5430db55bc282694fb9d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:33:50.390090 19905 raft_consensus.cc:2258] T 4e049c434d8d46f4a1b3d0ff08a37268 P 9eb2f1a7fb524990b88665f5acd19dbd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:33:50.390177 19906 tablet_replica.cc:330] T 4e049c434d8d46f4a1b3d0ff08a37268 P 029932492fa8492c80cc896ab4c881f6: stopping tablet replica
I20260810 22:33:50.390313 19906 raft_consensus.cc:2229] T 4e049c434d8d46f4a1b3d0ff08a37268 P 029932492fa8492c80cc896ab4c881f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:33:50.390480 19906 raft_consensus.cc:2258] T 4e049c434d8d46f4a1b3d0ff08a37268 P 029932492fa8492c80cc896ab4c881f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:33:50.390655 19227 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53126:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 22:33:50.390817 19909 ts_tablet_manager.cc:1905] T 4e049c434d8d46f4a1b3d0ff08a37268 P 5d3b49686ca5430db55bc282694fb9d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:33:50.390959 19905 ts_tablet_manager.cc:1905] T 4e049c434d8d46f4a1b3d0ff08a37268 P 9eb2f1a7fb524990b88665f5acd19dbd: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 22:33:50.391180 19227 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:33:50.393477 19909 ts_tablet_manager.cc:1918] T 4e049c434d8d46f4a1b3d0ff08a37268 P 5d3b49686ca5430db55bc282694fb9d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 22:33:50.393596 19909 log.cc:1196] T 4e049c434d8d46f4a1b3d0ff08a37268 P 5d3b49686ca5430db55bc282694fb9d4: Deleting WAL directory at /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/ts-0/wal/wals/4e049c434d8d46f4a1b3d0ff08a37268
I20260810 22:33:50.393918 19909 ts_tablet_manager.cc:1939] T 4e049c434d8d46f4a1b3d0ff08a37268 P 5d3b49686ca5430db55bc282694fb9d4: Deleting consensus metadata
I20260810 22:33:50.393996 19905 ts_tablet_manager.cc:1918] T 4e049c434d8d46f4a1b3d0ff08a37268 P 9eb2f1a7fb524990b88665f5acd19dbd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 22:33:50.394083 19905 log.cc:1196] T 4e049c434d8d46f4a1b3d0ff08a37268 P 9eb2f1a7fb524990b88665f5acd19dbd: Deleting WAL directory at /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/ts-1/wal/wals/4e049c434d8d46f4a1b3d0ff08a37268
I20260810 22:33:50.394713 19905 ts_tablet_manager.cc:1939] T 4e049c434d8d46f4a1b3d0ff08a37268 P 9eb2f1a7fb524990b88665f5acd19dbd: Deleting consensus metadata
I20260810 22:33:50.394873 19214 catalog_manager.cc:4841] TS 5d3b49686ca5430db55bc282694fb9d4 (127.16.233.65:43731): tablet 4e049c434d8d46f4a1b3d0ff08a37268 (table test-table [id=161bbe42111c4417859ebe5d13670645]) successfully deleted
I20260810 22:33:50.395614 19213 catalog_manager.cc:4841] TS 9eb2f1a7fb524990b88665f5acd19dbd (127.16.233.66:39973): tablet 4e049c434d8d46f4a1b3d0ff08a37268 (table test-table [id=161bbe42111c4417859ebe5d13670645]) successfully deleted
I20260810 22:33:50.396190 19906 ts_tablet_manager.cc:1905] T 4e049c434d8d46f4a1b3d0ff08a37268 P 029932492fa8492c80cc896ab4c881f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:33:50.397188 19381 tablet_service.cc:1467] Processing CreateTablet for tablet 37dc9717e04142589a7969791dad6a5f (DEFAULT_TABLE table=test-table [id=25af11615a6a4a98a8bdc619405b1da8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:33:50.397398 19692 tablet_service.cc:1467] Processing CreateTablet for tablet 37dc9717e04142589a7969791dad6a5f (DEFAULT_TABLE table=test-table [id=25af11615a6a4a98a8bdc619405b1da8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:33:50.397497 19381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37dc9717e04142589a7969791dad6a5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:33:50.397651 19692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37dc9717e04142589a7969791dad6a5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:33:50.398914 19906 ts_tablet_manager.cc:1918] T 4e049c434d8d46f4a1b3d0ff08a37268 P 029932492fa8492c80cc896ab4c881f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 22:33:50.399024 19906 log.cc:1196] T 4e049c434d8d46f4a1b3d0ff08a37268 P 029932492fa8492c80cc896ab4c881f6: Deleting WAL directory at /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/ts-2/wal/wals/4e049c434d8d46f4a1b3d0ff08a37268
I20260810 22:33:50.399310 19906 ts_tablet_manager.cc:1939] T 4e049c434d8d46f4a1b3d0ff08a37268 P 029932492fa8492c80cc896ab4c881f6: Deleting consensus metadata
I20260810 22:33:50.400071 19923 tablet_bootstrap.cc:492] T 37dc9717e04142589a7969791dad6a5f P 5d3b49686ca5430db55bc282694fb9d4: Bootstrap starting.
I20260810 22:33:50.400285 19920 tablet_bootstrap.cc:492] T 37dc9717e04142589a7969791dad6a5f P 029932492fa8492c80cc896ab4c881f6: Bootstrap starting.
I20260810 22:33:50.400319 19214 catalog_manager.cc:4841] TS 029932492fa8492c80cc896ab4c881f6 (127.16.233.67:40807): tablet 4e049c434d8d46f4a1b3d0ff08a37268 (table test-table [id=161bbe42111c4417859ebe5d13670645]) successfully deleted
I20260810 22:33:50.400516 19542 tablet_service.cc:1467] Processing CreateTablet for tablet 37dc9717e04142589a7969791dad6a5f (DEFAULT_TABLE table=test-table [id=25af11615a6a4a98a8bdc619405b1da8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:33:50.400804 19542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37dc9717e04142589a7969791dad6a5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:33:50.401305 19920 tablet_bootstrap.cc:654] T 37dc9717e04142589a7969791dad6a5f P 029932492fa8492c80cc896ab4c881f6: Neither blocks nor log segments found. Creating new log.
I20260810 22:33:50.401394 19923 tablet_bootstrap.cc:654] T 37dc9717e04142589a7969791dad6a5f P 5d3b49686ca5430db55bc282694fb9d4: Neither blocks nor log segments found. Creating new log.
I20260810 22:33:50.402426 19920 tablet_bootstrap.cc:492] T 37dc9717e04142589a7969791dad6a5f P 029932492fa8492c80cc896ab4c881f6: No bootstrap required, opened a new log
I20260810 22:33:50.402526 19920 ts_tablet_manager.cc:1397] T 37dc9717e04142589a7969791dad6a5f P 029932492fa8492c80cc896ab4c881f6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 22:33:50.402796 19923 tablet_bootstrap.cc:492] T 37dc9717e04142589a7969791dad6a5f P 5d3b49686ca5430db55bc282694fb9d4: No bootstrap required, opened a new log
I20260810 22:33:50.402922 19923 ts_tablet_manager.cc:1397] T 37dc9717e04142589a7969791dad6a5f P 5d3b49686ca5430db55bc282694fb9d4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 22:33:50.402918 19920 raft_consensus.cc:348] T 37dc9717e04142589a7969791dad6a5f P 029932492fa8492c80cc896ab4c881f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3b49686ca5430db55bc282694fb9d4" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 43731 } } peers { permanent_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39973 } } peers { permanent_uuid: "029932492fa8492c80cc896ab4c881f6" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 40807 } }
I20260810 22:33:50.403049 19920 raft_consensus.cc:374] T 37dc9717e04142589a7969791dad6a5f P 029932492fa8492c80cc896ab4c881f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:33:50.403100 19920 raft_consensus.cc:729] T 37dc9717e04142589a7969791dad6a5f P 029932492fa8492c80cc896ab4c881f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 029932492fa8492c80cc896ab4c881f6, State: Initialized, Role: FOLLOWER
I20260810 22:33:50.403223 19920 consensus_queue.cc:260] T 37dc9717e04142589a7969791dad6a5f P 029932492fa8492c80cc896ab4c881f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3b49686ca5430db55bc282694fb9d4" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 43731 } } peers { permanent_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39973 } } peers { permanent_uuid: "029932492fa8492c80cc896ab4c881f6" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 40807 } }
I20260810 22:33:50.403509 19920 ts_tablet_manager.cc:1428] T 37dc9717e04142589a7969791dad6a5f P 029932492fa8492c80cc896ab4c881f6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 22:33:50.403483 19923 raft_consensus.cc:348] T 37dc9717e04142589a7969791dad6a5f P 5d3b49686ca5430db55bc282694fb9d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3b49686ca5430db55bc282694fb9d4" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 43731 } } peers { permanent_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39973 } } peers { permanent_uuid: "029932492fa8492c80cc896ab4c881f6" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 40807 } }
I20260810 22:33:50.403627 19923 raft_consensus.cc:374] T 37dc9717e04142589a7969791dad6a5f P 5d3b49686ca5430db55bc282694fb9d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:33:50.403676 19923 raft_consensus.cc:729] T 37dc9717e04142589a7969791dad6a5f P 5d3b49686ca5430db55bc282694fb9d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d3b49686ca5430db55bc282694fb9d4, State: Initialized, Role: FOLLOWER
I20260810 22:33:50.403743 19919 tablet_bootstrap.cc:492] T 37dc9717e04142589a7969791dad6a5f P 9eb2f1a7fb524990b88665f5acd19dbd: Bootstrap starting.
I20260810 22:33:50.403916 19923 consensus_queue.cc:260] T 37dc9717e04142589a7969791dad6a5f P 5d3b49686ca5430db55bc282694fb9d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3b49686ca5430db55bc282694fb9d4" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 43731 } } peers { permanent_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39973 } } peers { permanent_uuid: "029932492fa8492c80cc896ab4c881f6" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 40807 } }
I20260810 22:33:50.404628 19919 tablet_bootstrap.cc:654] T 37dc9717e04142589a7969791dad6a5f P 9eb2f1a7fb524990b88665f5acd19dbd: Neither blocks nor log segments found. Creating new log.
I20260810 22:33:50.405938 19923 ts_tablet_manager.cc:1428] T 37dc9717e04142589a7969791dad6a5f P 5d3b49686ca5430db55bc282694fb9d4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260810 22:33:50.405964 19919 tablet_bootstrap.cc:492] T 37dc9717e04142589a7969791dad6a5f P 9eb2f1a7fb524990b88665f5acd19dbd: No bootstrap required, opened a new log
I20260810 22:33:50.406054 19919 ts_tablet_manager.cc:1397] T 37dc9717e04142589a7969791dad6a5f P 9eb2f1a7fb524990b88665f5acd19dbd: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260810 22:33:50.406507 19919 raft_consensus.cc:348] T 37dc9717e04142589a7969791dad6a5f P 9eb2f1a7fb524990b88665f5acd19dbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3b49686ca5430db55bc282694fb9d4" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 43731 } } peers { permanent_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39973 } } peers { permanent_uuid: "029932492fa8492c80cc896ab4c881f6" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 40807 } }
I20260810 22:33:50.406656 19919 raft_consensus.cc:374] T 37dc9717e04142589a7969791dad6a5f P 9eb2f1a7fb524990b88665f5acd19dbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:33:50.406689 19919 raft_consensus.cc:729] T 37dc9717e04142589a7969791dad6a5f P 9eb2f1a7fb524990b88665f5acd19dbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9eb2f1a7fb524990b88665f5acd19dbd, State: Initialized, Role: FOLLOWER
I20260810 22:33:50.406818 19919 consensus_queue.cc:260] T 37dc9717e04142589a7969791dad6a5f P 9eb2f1a7fb524990b88665f5acd19dbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3b49686ca5430db55bc282694fb9d4" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 43731 } } peers { permanent_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39973 } } peers { permanent_uuid: "029932492fa8492c80cc896ab4c881f6" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 40807 } }
I20260810 22:33:50.407105 19919 ts_tablet_manager.cc:1428] T 37dc9717e04142589a7969791dad6a5f P 9eb2f1a7fb524990b88665f5acd19dbd: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 22:33:50.535528 19926 raft_consensus.cc:482] T 37dc9717e04142589a7969791dad6a5f P 5d3b49686ca5430db55bc282694fb9d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:33:50.535709 19926 raft_consensus.cc:504] T 37dc9717e04142589a7969791dad6a5f P 5d3b49686ca5430db55bc282694fb9d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3b49686ca5430db55bc282694fb9d4" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 43731 } } peers { permanent_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39973 } } peers { permanent_uuid: "029932492fa8492c80cc896ab4c881f6" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 40807 } }
I20260810 22:33:50.536283 19926 leader_election.cc:288] T 37dc9717e04142589a7969791dad6a5f P 5d3b49686ca5430db55bc282694fb9d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9eb2f1a7fb524990b88665f5acd19dbd (127.16.233.66:39973), 029932492fa8492c80cc896ab4c881f6 (127.16.233.67:40807)
I20260810 22:33:50.536788 19715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37dc9717e04142589a7969791dad6a5f" candidate_uuid: "5d3b49686ca5430db55bc282694fb9d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "029932492fa8492c80cc896ab4c881f6" is_pre_election: true
I20260810 22:33:50.536783 19563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37dc9717e04142589a7969791dad6a5f" candidate_uuid: "5d3b49686ca5430db55bc282694fb9d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" is_pre_election: true
I20260810 22:33:50.536988 19715 raft_consensus.cc:2454] T 37dc9717e04142589a7969791dad6a5f P 029932492fa8492c80cc896ab4c881f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d3b49686ca5430db55bc282694fb9d4 in term 0.
I20260810 22:33:50.537005 19563 raft_consensus.cc:2454] T 37dc9717e04142589a7969791dad6a5f P 9eb2f1a7fb524990b88665f5acd19dbd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d3b49686ca5430db55bc282694fb9d4 in term 0.
I20260810 22:33:50.537353 19326 leader_election.cc:302] T 37dc9717e04142589a7969791dad6a5f P 5d3b49686ca5430db55bc282694fb9d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d3b49686ca5430db55bc282694fb9d4, 9eb2f1a7fb524990b88665f5acd19dbd; no voters:
I20260810 22:33:50.537549 19926 raft_consensus.cc:2781] T 37dc9717e04142589a7969791dad6a5f P 5d3b49686ca5430db55bc282694fb9d4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:33:50.537617 19926 raft_consensus.cc:482] T 37dc9717e04142589a7969791dad6a5f P 5d3b49686ca5430db55bc282694fb9d4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:33:50.537662 19926 raft_consensus.cc:3037] T 37dc9717e04142589a7969791dad6a5f P 5d3b49686ca5430db55bc282694fb9d4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:33:50.538786 19926 raft_consensus.cc:504] T 37dc9717e04142589a7969791dad6a5f P 5d3b49686ca5430db55bc282694fb9d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3b49686ca5430db55bc282694fb9d4" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 43731 } } peers { permanent_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39973 } } peers { permanent_uuid: "029932492fa8492c80cc896ab4c881f6" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 40807 } }
I20260810 22:33:50.539209 19926 leader_election.cc:288] T 37dc9717e04142589a7969791dad6a5f P 5d3b49686ca5430db55bc282694fb9d4 [CANDIDATE]: Term 1 election: Requested vote from peers 9eb2f1a7fb524990b88665f5acd19dbd (127.16.233.66:39973), 029932492fa8492c80cc896ab4c881f6 (127.16.233.67:40807)
I20260810 22:33:50.539747 19715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37dc9717e04142589a7969791dad6a5f" candidate_uuid: "5d3b49686ca5430db55bc282694fb9d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "029932492fa8492c80cc896ab4c881f6"
I20260810 22:33:50.539705 19563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37dc9717e04142589a7969791dad6a5f" candidate_uuid: "5d3b49686ca5430db55bc282694fb9d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9eb2f1a7fb524990b88665f5acd19dbd"
I20260810 22:33:50.539934 19563 raft_consensus.cc:3037] T 37dc9717e04142589a7969791dad6a5f P 9eb2f1a7fb524990b88665f5acd19dbd [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:33:50.539920 19715 raft_consensus.cc:3037] T 37dc9717e04142589a7969791dad6a5f P 029932492fa8492c80cc896ab4c881f6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:33:50.541343 19715 raft_consensus.cc:2454] T 37dc9717e04142589a7969791dad6a5f P 029932492fa8492c80cc896ab4c881f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d3b49686ca5430db55bc282694fb9d4 in term 1.
I20260810 22:33:50.541342 19563 raft_consensus.cc:2454] T 37dc9717e04142589a7969791dad6a5f P 9eb2f1a7fb524990b88665f5acd19dbd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d3b49686ca5430db55bc282694fb9d4 in term 1.
I20260810 22:33:50.541743 19327 leader_election.cc:302] T 37dc9717e04142589a7969791dad6a5f P 5d3b49686ca5430db55bc282694fb9d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 029932492fa8492c80cc896ab4c881f6, 5d3b49686ca5430db55bc282694fb9d4; no voters:
I20260810 22:33:50.542001 19926 raft_consensus.cc:2781] T 37dc9717e04142589a7969791dad6a5f P 5d3b49686ca5430db55bc282694fb9d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:33:50.542268 19926 raft_consensus.cc:686] T 37dc9717e04142589a7969791dad6a5f P 5d3b49686ca5430db55bc282694fb9d4 [term 1 LEADER]: Becoming Leader. State: Replica: 5d3b49686ca5430db55bc282694fb9d4, State: Running, Role: LEADER
I20260810 22:33:50.542616 19926 consensus_queue.cc:237] T 37dc9717e04142589a7969791dad6a5f P 5d3b49686ca5430db55bc282694fb9d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3b49686ca5430db55bc282694fb9d4" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 43731 } } peers { permanent_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39973 } } peers { permanent_uuid: "029932492fa8492c80cc896ab4c881f6" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 40807 } }
I20260810 22:33:50.545276 19228 catalog_manager.cc:5495] T 37dc9717e04142589a7969791dad6a5f P 5d3b49686ca5430db55bc282694fb9d4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d3b49686ca5430db55bc282694fb9d4 (127.16.233.65). New cstate: current_term: 1 leader_uuid: "5d3b49686ca5430db55bc282694fb9d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d3b49686ca5430db55bc282694fb9d4" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 43731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "029932492fa8492c80cc896ab4c881f6" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 40807 } health_report { overall_health: UNKNOWN } } }
I20260810 22:33:50.573386 19938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:50.569240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42434 (local address 127.16.233.65:43731)
0810 22:33:50.569593 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:50.569600 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:50.569614 (+ 14us) server_negotiation.cc:407] Connection header received
0810 22:33:50.569694 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:50.569697 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:50.569752 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:50.569928 (+ 176us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:33:50.570163 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:50.570703 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:50.571647 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:50.571950 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:50.572041 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:50.572317 (+ 276us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:33:50.572794 (+ 477us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:33:50.573027 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:50.573139 (+ 112us) server_negotiation.cc:299] Negotiation successful
0810 22:33:50.573205 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":189,"threads_started":1}
I20260810 22:33:50.585489 19929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:50.581560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:45235 (local address 127.16.233.66:39973)
0810 22:33:50.581766 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:50.581774 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:50.582081 (+ 307us) server_negotiation.cc:407] Connection header received
0810 22:33:50.582301 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:50.582305 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:50.582362 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:50.582443 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:33:50.582740 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:50.583438 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:50.584341 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:50.584578 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:50.584669 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:50.585078 (+ 409us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:33:50.585158 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:33:50.585221 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:50.585303 (+ 82us) server_negotiation.cc:299] Negotiation successful
0810 22:33:50.585361 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260810 22:33:50.585512 19941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:50.581432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.66:39973 (local address 127.16.233.65:45235)
0810 22:33:50.581968 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:50.581986 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:50.582134 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:50.582468 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:50.582470 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:50.582477 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 22:33:50.582608 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:50.582613 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:50.583578 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:50.583582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:50.584221 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:50.584230 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:50.584703 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:50.584707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:50.584789 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:50.584937 (+ 148us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 22:33:50.585236 (+ 299us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 22:33:50.585241 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 22:33:50.585291 (+ 50us) client_negotiation.cc:240] Negotiation successful
0810 22:33:50.585365 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":176,"threads_started":1}
I20260810 22:33:50.593231 19563 raft_consensus.cc:1261] T 37dc9717e04142589a7969791dad6a5f P 9eb2f1a7fb524990b88665f5acd19dbd [term 1 FOLLOWER]: Refusing update from remote peer 5d3b49686ca5430db55bc282694fb9d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:33:50.593223 19715 raft_consensus.cc:1261] T 37dc9717e04142589a7969791dad6a5f P 029932492fa8492c80cc896ab4c881f6 [term 1 FOLLOWER]: Refusing update from remote peer 5d3b49686ca5430db55bc282694fb9d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:33:50.593885 19936 consensus_queue.cc:1035] T 37dc9717e04142589a7969791dad6a5f P 5d3b49686ca5430db55bc282694fb9d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9eb2f1a7fb524990b88665f5acd19dbd" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:33:50.594158 19926 consensus_queue.cc:1035] T 37dc9717e04142589a7969791dad6a5f P 5d3b49686ca5430db55bc282694fb9d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "029932492fa8492c80cc896ab4c881f6" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:33:50.615209 19938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:50.611391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:47871 (local address 127.16.233.65:43731)
0810 22:33:50.611558 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:50.611563 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:50.611580 (+ 17us) server_negotiation.cc:407] Connection header received
0810 22:33:50.611631 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:50.611634 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:50.611701 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:50.611782 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:33:50.612074 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:50.612760 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:50.613694 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:50.613998 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:50.614139 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:50.614561 (+ 422us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:33:50.614659 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:33:50.614742 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:50.614986 (+ 244us) server_negotiation.cc:299] Negotiation successful
0810 22:33:50.615064 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 22:33:50.615180 19948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:50.611081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.65:43731 (local address 127.16.233.66:47871)
0810 22:33:50.611380 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:50.611397 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:50.611502 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:50.611811 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:50.611814 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:50.611822 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 22:33:50.611952 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:50.611958 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:50.612894 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:50.612898 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:50.613566 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:50.613576 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:50.614182 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:50.614186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:50.614340 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:50.614386 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 22:33:50.614781 (+ 395us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 22:33:50.614791 (+ 10us) client_negotiation.cc:769] Sending connection context
0810 22:33:50.614883 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 22:33:50.614989 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":130,"threads_started":1}
W20260810 22:33:50.616119 19884 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260810 22:33:52.181154 19224 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53126:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:33:52.181331 19224 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53126:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:33:52.184121 19224 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5652ecc5dafa4979ab57ef1506d00660: Sending DeleteTablet for 3 replicas of tablet 37dc9717e04142589a7969791dad6a5f
I20260810 22:33:52.184753 19381 tablet_service.cc:1514] Processing DeleteTablet for tablet 37dc9717e04142589a7969791dad6a5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:33:52 UTC) from {username='kudu', principal='kudu/127.16.233.126@KRBTEST.COM'} at 127.0.0.1:36462
I20260810 22:33:52.184845 19692 tablet_service.cc:1514] Processing DeleteTablet for tablet 37dc9717e04142589a7969791dad6a5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:33:52 UTC) from {username='kudu', principal='kudu/127.16.233.126@KRBTEST.COM'} at 127.0.0.1:53392
I20260810 22:33:52.184952 19542 tablet_service.cc:1514] Processing DeleteTablet for tablet 37dc9717e04142589a7969791dad6a5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:33:52 UTC) from {username='kudu', principal='kudu/127.16.233.126@KRBTEST.COM'} at 127.0.0.1:58396
I20260810 22:33:52.185303 19975 tablet_replica.cc:330] T 37dc9717e04142589a7969791dad6a5f P 029932492fa8492c80cc896ab4c881f6: stopping tablet replica
I20260810 22:33:52.185361 19976 tablet_replica.cc:330] T 37dc9717e04142589a7969791dad6a5f P 9eb2f1a7fb524990b88665f5acd19dbd: stopping tablet replica
I20260810 22:33:52.185386 19974 tablet_replica.cc:330] T 37dc9717e04142589a7969791dad6a5f P 5d3b49686ca5430db55bc282694fb9d4: stopping tablet replica
I20260810 22:33:52.185503 19974 raft_consensus.cc:2229] T 37dc9717e04142589a7969791dad6a5f P 5d3b49686ca5430db55bc282694fb9d4 [term 1 LEADER]: Raft consensus shutting down.
I20260810 22:33:52.185527 19976 raft_consensus.cc:2229] T 37dc9717e04142589a7969791dad6a5f P 9eb2f1a7fb524990b88665f5acd19dbd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:33:52.185647 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 19306
I20260810 22:33:52.185683 19976 raft_consensus.cc:2258] T 37dc9717e04142589a7969791dad6a5f P 9eb2f1a7fb524990b88665f5acd19dbd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:33:52.188347 19976 ts_tablet_manager.cc:1905] T 37dc9717e04142589a7969791dad6a5f P 9eb2f1a7fb524990b88665f5acd19dbd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:33:52.192523 19976 ts_tablet_manager.cc:1918] T 37dc9717e04142589a7969791dad6a5f P 9eb2f1a7fb524990b88665f5acd19dbd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 22:33:52.192638 19976 log.cc:1196] T 37dc9717e04142589a7969791dad6a5f P 9eb2f1a7fb524990b88665f5acd19dbd: Deleting WAL directory at /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/ts-1/wal/wals/37dc9717e04142589a7969791dad6a5f
I20260810 22:33:52.185653 19975 raft_consensus.cc:2229] T 37dc9717e04142589a7969791dad6a5f P 029932492fa8492c80cc896ab4c881f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:33:52.192957 19975 raft_consensus.cc:2258] T 37dc9717e04142589a7969791dad6a5f P 029932492fa8492c80cc896ab4c881f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:33:52.192997 19976 ts_tablet_manager.cc:1939] T 37dc9717e04142589a7969791dad6a5f P 9eb2f1a7fb524990b88665f5acd19dbd: Deleting consensus metadata
I20260810 22:33:52.193672 19975 ts_tablet_manager.cc:1905] T 37dc9717e04142589a7969791dad6a5f P 029932492fa8492c80cc896ab4c881f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:33:52.193993 19213 catalog_manager.cc:4841] TS 9eb2f1a7fb524990b88665f5acd19dbd (127.16.233.66:39973): tablet 37dc9717e04142589a7969791dad6a5f (table test-table [id=25af11615a6a4a98a8bdc619405b1da8]) successfully deleted
I20260810 22:33:52.195935 19975 ts_tablet_manager.cc:1918] T 37dc9717e04142589a7969791dad6a5f P 029932492fa8492c80cc896ab4c881f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 22:33:52.196034 19975 log.cc:1196] T 37dc9717e04142589a7969791dad6a5f P 029932492fa8492c80cc896ab4c881f6: Deleting WAL directory at /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786401215402979-17317-0/minicluster-data/ts-2/wal/wals/37dc9717e04142589a7969791dad6a5f
I20260810 22:33:52.196291 19975 ts_tablet_manager.cc:1939] T 37dc9717e04142589a7969791dad6a5f P 029932492fa8492c80cc896ab4c881f6: Deleting consensus metadata
I20260810 22:33:52.197115 19214 catalog_manager.cc:4841] TS 029932492fa8492c80cc896ab4c881f6 (127.16.233.67:40807): tablet 37dc9717e04142589a7969791dad6a5f (table test-table [id=25af11615a6a4a98a8bdc619405b1da8]) successfully deleted
W20260810 22:33:52.197438 19214 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.233.65:43731 (error 108)
I20260810 22:33:52.197692 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 19471
I20260810 22:33:52.201673 19980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:52.200848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.65:43731 (local address 127.0.0.1:42440)
0810 22:33:52.201425 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:52.201530 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.65:43731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":249,"threads_started":1}
W20260810 22:33:52.201809 19214 catalog_manager.cc:4584] TS 5d3b49686ca5430db55bc282694fb9d4 (127.16.233.65:43731): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 37dc9717e04142589a7969791dad6a5f: Network error: Client connection negotiation failed: client connection to 127.16.233.65:43731: connect: Connection refused (error 111)
I20260810 22:33:52.209220 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 19620
I20260810 22:33:52.218534 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 19192
2026-08-10T22:33:52Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (7194 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:33:52 dist-test-slave-77v9 krb5kdc[19983](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:33:52 dist-test-slave-77v9 krb5kdc[19983](info): set up 2 sockets
Aug 10 22:33:52 dist-test-slave-77v9 krb5kdc[19983](info): commencing operation
krb5kdc: starting...
W20260810 22:33:53.409228 17317 mini_kdc.cc:121] Time spent starting KDC: real 1.163s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Aug 10 22:33:53 dist-test-slave-77v9 krb5kdc[19983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:33:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:33:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.233.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab.
Entry for principal kudu/127.16.233.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab.
I20260810 22:33:53.557497 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:38957
--webserver_interface=127.16.233.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab
--principal=kudu/127.16.233.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:41351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.126:38957
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:33:53.684787 19998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:33:53.685138 19998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:33:53.685184 19998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:33:53.689841 19998 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:33:53.689936 19998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:33:53.689956 19998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:33:53.689975 19998 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:33:53.689990 19998 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:33:53.696226 19998 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:41351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.126:38957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab
--principal=kudu/127.16.233.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:38957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19998
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:33:53.697556 19998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:33:53.698765 19998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:33:53.709475 20008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:33:53.709606 19998 server_base.cc:1034] running on GCE node
W20260810 22:33:53.709856 20011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:33:53.710249 20009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:33:53.711027 19998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:33:53.712522 19998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:33:53.713953 19998 hybrid_clock.cc:648] HybridClock initialized: now 1786401233713936 us; error 69 us; skew 500 ppm
Aug 10 22:33:53 dist-test-slave-77v9 krb5kdc[19983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401233, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:33:53.717614 19998 init.cc:377] Logged in from keytab as kudu/127.16.233.126@KRBTEST.COM (short username kudu)
I20260810 22:33:53.718896 19998 webserver.cc:460] Webserver started at http://127.16.233.126:37231/ using document root <none> and password file <none>
I20260810 22:33:53.719547 19998 fs_manager.cc:359] Metadata directory not provided
I20260810 22:33:53.719651 19998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:33:53.719945 19998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:33:53.722498 19998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/minicluster-data/master-0/data/instance:
uuid: "f773248938524efcb2972d3227a07bd3"
format_stamp: "Formatted at 2026-08-10 22:33:53 on dist-test-slave-77v9"
server_key: "81016581397bad373049080f6b0f5ea3"
server_key_iv: "39ce41eeb6b66e71eb51d2013478ef9e"
server_key_version: "encryptionkey@0"
I20260810 22:33:53.723209 19998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/minicluster-data/master-0/wal/instance:
uuid: "f773248938524efcb2972d3227a07bd3"
format_stamp: "Formatted at 2026-08-10 22:33:53 on dist-test-slave-77v9"
server_key: "81016581397bad373049080f6b0f5ea3"
server_key_iv: "39ce41eeb6b66e71eb51d2013478ef9e"
server_key_version: "encryptionkey@0"
I20260810 22:33:53.727677 19998 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 22:33:53.730746 20019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:53.732219 19998 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 22:33:53.732450 19998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/minicluster-data/master-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/minicluster-data/master-0/wal
uuid: "f773248938524efcb2972d3227a07bd3"
format_stamp: "Formatted at 2026-08-10 22:33:53 on dist-test-slave-77v9"
server_key: "81016581397bad373049080f6b0f5ea3"
server_key_iv: "39ce41eeb6b66e71eb51d2013478ef9e"
server_key_version: "encryptionkey@0"
I20260810 22:33:53.732609 19998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:33:53.757382 19998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:33:53.760926 19998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:33:53.761173 19998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:33:53.769287 19998 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.126:38957
I20260810 22:33:53.769347 20076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.126:38957 every 8 connection(s)
I20260810 22:33:53.770403 19998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
I20260810 22:33:53.773603 20077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:33:53.776682 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 19998
I20260810 22:33:53.776870 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/minicluster-data/master-0/wal/instance
I20260810 22:33:53.777166 17317 external_mini_cluster.cc:1428] Setting key ab2b4fab1351871d1a63222541257489
I20260810 22:33:53.779932 20077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f773248938524efcb2972d3227a07bd3: Bootstrap starting.
I20260810 22:33:53.782694 20077 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f773248938524efcb2972d3227a07bd3: Neither blocks nor log segments found. Creating new log.
I20260810 22:33:53.783545 20077 log.cc:824] T 00000000000000000000000000000000 P f773248938524efcb2972d3227a07bd3: Log is configured to *not* fsync() on all Append() calls
I20260810 22:33:53.786165 20077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f773248938524efcb2972d3227a07bd3: No bootstrap required, opened a new log
Aug 10 22:33:53 dist-test-slave-77v9 krb5kdc[19983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:33:53.790283 20077 raft_consensus.cc:348] T 00000000000000000000000000000000 P f773248938524efcb2972d3227a07bd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f773248938524efcb2972d3227a07bd3" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 38957 } }
I20260810 22:33:53.790553 20077 raft_consensus.cc:374] T 00000000000000000000000000000000 P f773248938524efcb2972d3227a07bd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:33:53.790589 20077 raft_consensus.cc:729] T 00000000000000000000000000000000 P f773248938524efcb2972d3227a07bd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f773248938524efcb2972d3227a07bd3, State: Initialized, Role: FOLLOWER
I20260810 22:33:53.791143 20077 consensus_queue.cc:260] T 00000000000000000000000000000000 P f773248938524efcb2972d3227a07bd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f773248938524efcb2972d3227a07bd3" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 38957 } }
I20260810 22:33:53.791332 20077 raft_consensus.cc:388] T 00000000000000000000000000000000 P f773248938524efcb2972d3227a07bd3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:33:53.791414 20077 raft_consensus.cc:482] T 00000000000000000000000000000000 P f773248938524efcb2972d3227a07bd3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:33:53.791513 20077 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f773248938524efcb2972d3227a07bd3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:33:53.792740 20077 raft_consensus.cc:504] T 00000000000000000000000000000000 P f773248938524efcb2972d3227a07bd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f773248938524efcb2972d3227a07bd3" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 38957 } }
I20260810 22:33:53.793210 20077 leader_election.cc:302] T 00000000000000000000000000000000 P f773248938524efcb2972d3227a07bd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f773248938524efcb2972d3227a07bd3; no voters:
I20260810 22:33:53.793632 20077 leader_election.cc:288] T 00000000000000000000000000000000 P f773248938524efcb2972d3227a07bd3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:33:53.793870 20082 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f773248938524efcb2972d3227a07bd3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:33:53.794144 20082 raft_consensus.cc:686] T 00000000000000000000000000000000 P f773248938524efcb2972d3227a07bd3 [term 1 LEADER]: Becoming Leader. State: Replica: f773248938524efcb2972d3227a07bd3, State: Running, Role: LEADER
I20260810 22:33:53.794595 20082 consensus_queue.cc:237] T 00000000000000000000000000000000 P f773248938524efcb2972d3227a07bd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f773248938524efcb2972d3227a07bd3" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 38957 } }
I20260810 22:33:53.795042 20080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:53.778749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47490 (local address 127.16.233.126:38957)
0810 22:33:53.779260 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:53.779276 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:53.779313 (+ 37us) server_negotiation.cc:407] Connection header received
0810 22:33:53.779947 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:53.779963 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:53.780361 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:53.780649 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:53.781680 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:53.782749 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:53.783706 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:53.784230 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:53.784441 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:53.787592 (+ 3151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:53.787600 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:53.787616 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:53.787654 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:53.790791 (+ 3137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:53.791569 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:53.791581 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:53.791587 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:53.791767 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:53.792143 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:53.792146 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:53.792148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:53.792691 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:53.793031 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:53.793391 (+ 360us) server_negotiation.cc:299] Negotiation successful
0810 22:33:53.793794 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":209,"threads_started":1}
I20260810 22:33:53.795094 20077 sys_catalog.cc:564] T 00000000000000000000000000000000 P f773248938524efcb2972d3227a07bd3 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:33:53.796032 20082 sys_catalog.cc:455] T 00000000000000000000000000000000 P f773248938524efcb2972d3227a07bd3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f773248938524efcb2972d3227a07bd3. Latest consensus state: current_term: 1 leader_uuid: "f773248938524efcb2972d3227a07bd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f773248938524efcb2972d3227a07bd3" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 38957 } } }
I20260810 22:33:53.796289 20082 sys_catalog.cc:458] T 00000000000000000000000000000000 P f773248938524efcb2972d3227a07bd3 [sys.catalog]: This master's current role is: LEADER
I20260810 22:33:53.796545 20083 sys_catalog.cc:455] T 00000000000000000000000000000000 P f773248938524efcb2972d3227a07bd3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f773248938524efcb2972d3227a07bd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f773248938524efcb2972d3227a07bd3" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 38957 } } }
I20260810 22:33:53.796658 20083 sys_catalog.cc:458] T 00000000000000000000000000000000 P f773248938524efcb2972d3227a07bd3 [sys.catalog]: This master's current role is: LEADER
I20260810 22:33:53.796753 20093 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:33:53.800155 20093 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:33:53.806640 20093 catalog_manager.cc:1347] Generated new cluster ID: 3b7b5e334ca1434dbd499a154259fb0e
I20260810 22:33:53.806741 20093 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:33:53.828706 20093 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:33:53.830127 20093 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:33:53.840974 20093 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f773248938524efcb2972d3227a07bd3: Generated new TSK 0
I20260810 22:33:53.841895 20093 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.233.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab.
Entry for principal kudu/127.16.233.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab.
I20260810 22:33:53.884534 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.65:0
--local_ip_for_outbound_sockets=127.16.233.65
--webserver_interface=127.16.233.65
--webserver_port=0
--tserver_master_addrs=127.16.233.126:38957
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab
--principal=kudu/127.16.233.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:41351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:33:54.008215 20105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:33:54.008525 20105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:33:54.008572 20105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:33:54.012997 20105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:33:54.013077 20105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:33:54.013140 20105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.65
I20260810 22:33:54.018778 20105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:41351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab
--principal=kudu/127.16.233.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.233.65
--webserver_port=0
--tserver_master_addrs=127.16.233.126:38957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.65
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:33:54.020040 20105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:33:54.021209 20105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:33:54.032236 20110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:33:54.032226 20111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:33:54.032575 20105 server_base.cc:1034] running on GCE node
W20260810 22:33:54.032218 20113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:33:54.033296 20105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:33:54.034067 20105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:33:54.035307 20105 hybrid_clock.cc:648] HybridClock initialized: now 1786401234035287 us; error 36 us; skew 500 ppm
Aug 10 22:33:54 dist-test-slave-77v9 krb5kdc[19983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401234, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:33:54.038892 20105 init.cc:377] Logged in from keytab as kudu/127.16.233.65@KRBTEST.COM (short username kudu)
I20260810 22:33:54.039990 20105 webserver.cc:460] Webserver started at http://127.16.233.65:39507/ using document root <none> and password file <none>
I20260810 22:33:54.040637 20105 fs_manager.cc:359] Metadata directory not provided
I20260810 22:33:54.040743 20105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:33:54.041043 20105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:33:54.043512 20105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-0/data/instance:
uuid: "2a98ec8b454f46888137ed91c575ac37"
format_stamp: "Formatted at 2026-08-10 22:33:54 on dist-test-slave-77v9"
server_key: "0e4cc056adc40692a83777824da889df"
server_key_iv: "5ae2c408b468613c6ccf402ebe445218"
server_key_version: "encryptionkey@0"
I20260810 22:33:54.044162 20105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-0/wal/instance:
uuid: "2a98ec8b454f46888137ed91c575ac37"
format_stamp: "Formatted at 2026-08-10 22:33:54 on dist-test-slave-77v9"
server_key: "0e4cc056adc40692a83777824da889df"
server_key_iv: "5ae2c408b468613c6ccf402ebe445218"
server_key_version: "encryptionkey@0"
I20260810 22:33:54.048699 20105 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260810 22:33:54.052425 20120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:54.054157 20105 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260810 22:33:54.054292 20105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-0/wal
uuid: "2a98ec8b454f46888137ed91c575ac37"
format_stamp: "Formatted at 2026-08-10 22:33:54 on dist-test-slave-77v9"
server_key: "0e4cc056adc40692a83777824da889df"
server_key_iv: "5ae2c408b468613c6ccf402ebe445218"
server_key_version: "encryptionkey@0"
I20260810 22:33:54.054469 20105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:33:54.069468 20105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:33:54.073314 20105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:33:54.073560 20105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:33:54.074250 20105 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:33:54.075174 20105 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:33:54.075225 20105 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:54.075297 20105 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:33:54.075330 20105 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:54.085074 20105 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.65:34869
I20260810 22:33:54.085117 20242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.65:34869 every 8 connection(s)
I20260810 22:33:54.086109 20105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
I20260810 22:33:54.093688 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 20105
I20260810 22:33:54.093819 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-0/wal/instance
I20260810 22:33:54.094135 17317 external_mini_cluster.cc:1428] Setting key 2466ea7c87ee2cb8821d5da86782a3f5
Aug 10 22:33:54 dist-test-slave-77v9 krb5kdc[19983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401234, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.65@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:33:54.102535 20080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:54.088177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:35081 (local address 127.16.233.126:38957)
0810 22:33:54.088386 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:54.088391 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:54.089504 (+ 1113us) server_negotiation.cc:407] Connection header received
0810 22:33:54.090413 (+ 909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:54.090416 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:54.090469 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:54.090547 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:54.091804 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:54.092378 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:54.093691 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:54.094085 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:54.094206 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:54.097867 (+ 3661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:54.097876 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:54.097883 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:54.097924 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:54.099731 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:54.100565 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:54.100569 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:54.100570 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:54.100661 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:54.101148 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:54.101151 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:54.101152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:54.101369 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:54.101456 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:54.102250 (+ 794us) server_negotiation.cc:299] Negotiation successful
0810 22:33:54.102389 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260810 22:33:54.103951 20245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:54.088612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:38957 (local address 127.16.233.65:35081)
0810 22:33:54.089322 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:54.089375 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:54.090206 (+ 831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:54.090715 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:54.090722 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:54.091154 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:33:54.091663 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:54.091677 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:54.092498 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:54.092501 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:54.093457 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:54.093466 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:54.094283 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:54.094287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:54.094451 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:54.095325 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:33:54.095376 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:33:54.097626 (+ 2250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:33:54.099880 (+ 2254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:54.099889 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:54.099915 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:54.100422 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:54.100763 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:54.100769 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:54.100774 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:54.101020 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:54.101489 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:33:54.101500 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:33:54.101910 (+ 410us) client_negotiation.cc:769] Sending connection context
0810 22:33:54.102161 (+ 251us) client_negotiation.cc:240] Negotiation successful
0810 22:33:54.102608 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":142,"threads_started":1}
I20260810 22:33:54.105578 20243 heartbeater.cc:347] Connected to a master server at 127.16.233.126:38957
I20260810 22:33:54.105871 20243 heartbeater.cc:464] Registering TS with master...
I20260810 22:33:54.106629 20243 heartbeater.cc:511] Master 127.16.233.126:38957 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.16.233.66@KRBTEST.COM; defaulting to no policy
I20260810 22:33:54.108533 20037 ts_manager.cc:194] Registered new tserver with Master: 2a98ec8b454f46888137ed91c575ac37 (127.16.233.65:34869)
I20260810 22:33:54.109997 20037 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.65@KRBTEST.COM'} at 127.16.233.65:35081
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab.
Entry for principal kudu/127.16.233.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab.
I20260810 22:33:54.130455 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.66:0
--local_ip_for_outbound_sockets=127.16.233.66
--webserver_interface=127.16.233.66
--webserver_port=0
--tserver_master_addrs=127.16.233.126:38957
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab
--principal=kudu/127.16.233.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:41351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:33:54.250855 20248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:33:54.251159 20248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:33:54.251205 20248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:33:54.255425 20248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:33:54.255508 20248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:33:54.255571 20248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.66
I20260810 22:33:54.261770 20248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:41351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab
--principal=kudu/127.16.233.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.233.66
--webserver_port=0
--tserver_master_addrs=127.16.233.126:38957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.66
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:33:54.262953 20248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:33:54.264030 20248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:33:54.274886 20254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:33:54.275007 20248 server_base.cc:1034] running on GCE node
W20260810 22:33:54.274969 20256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:33:54.274886 20253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:33:54.275585 20248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:33:54.276340 20248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:33:54.277527 20248 hybrid_clock.cc:648] HybridClock initialized: now 1786401234277512 us; error 31 us; skew 500 ppm
Aug 10 22:33:54 dist-test-slave-77v9 krb5kdc[19983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401234, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:33:54.281134 20248 init.cc:377] Logged in from keytab as kudu/127.16.233.66@KRBTEST.COM (short username kudu)
I20260810 22:33:54.282337 20248 webserver.cc:460] Webserver started at http://127.16.233.66:35233/ using document root <none> and password file <none>
I20260810 22:33:54.283006 20248 fs_manager.cc:359] Metadata directory not provided
I20260810 22:33:54.283084 20248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:33:54.283355 20248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:33:54.285859 20248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-1/data/instance:
uuid: "826d60aa26564e539d87037cca08c8b3"
format_stamp: "Formatted at 2026-08-10 22:33:54 on dist-test-slave-77v9"
server_key: "668c6ea38f40d8b238b8f39b7ca20b04"
server_key_iv: "edb05bfbd2d255f06a54af385cfbf668"
server_key_version: "encryptionkey@0"
I20260810 22:33:54.286634 20248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-1/wal/instance:
uuid: "826d60aa26564e539d87037cca08c8b3"
format_stamp: "Formatted at 2026-08-10 22:33:54 on dist-test-slave-77v9"
server_key: "668c6ea38f40d8b238b8f39b7ca20b04"
server_key_iv: "edb05bfbd2d255f06a54af385cfbf668"
server_key_version: "encryptionkey@0"
I20260810 22:33:54.291427 20248 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 22:33:54.294935 20263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:54.297036 20248 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 22:33:54.297212 20248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-1/wal
uuid: "826d60aa26564e539d87037cca08c8b3"
format_stamp: "Formatted at 2026-08-10 22:33:54 on dist-test-slave-77v9"
server_key: "668c6ea38f40d8b238b8f39b7ca20b04"
server_key_iv: "edb05bfbd2d255f06a54af385cfbf668"
server_key_version: "encryptionkey@0"
I20260810 22:33:54.297372 20248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:33:54.315816 20248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:33:54.319830 20248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:33:54.320072 20248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:33:54.320777 20248 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:33:54.321782 20248 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:33:54.321870 20248 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:54.321941 20248 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:33:54.321991 20248 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:54.332374 20248 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.66:41585
I20260810 22:33:54.332425 20389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.66:41585 every 8 connection(s)
I20260810 22:33:54.333478 20248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
I20260810 22:33:54.339421 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 20248
I20260810 22:33:54.339568 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-1/wal/instance
I20260810 22:33:54.339859 17317 external_mini_cluster.cc:1428] Setting key 4ca64489a56af2981292d9b15688212e
Aug 10 22:33:54 dist-test-slave-77v9 krb5kdc[19983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401234, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.66@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:33:54.351089 20080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:54.335663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:52503 (local address 127.16.233.126:38957)
0810 22:33:54.335842 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:54.335846 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:54.336855 (+ 1009us) server_negotiation.cc:407] Connection header received
0810 22:33:54.338512 (+ 1657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:54.338515 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:54.338568 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:54.338649 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:54.340393 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:54.341160 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:54.342877 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:54.343125 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:54.343221 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:54.346502 (+ 3281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:54.346506 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:54.346508 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:54.346537 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:54.348259 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:54.349120 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:54.349123 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:54.349124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:54.349215 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:54.349704 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:54.349707 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:54.349708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:54.349946 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:54.350031 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:54.350813 (+ 782us) server_negotiation.cc:299] Negotiation successful
0810 22:33:54.350954 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 22:33:54.352432 20393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:54.336063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:38957 (local address 127.16.233.66:52503)
0810 22:33:54.336666 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:54.336718 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:54.338256 (+ 1538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:54.338835 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:54.338843 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:54.339369 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:33:54.340150 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:54.340172 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:54.341315 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:54.341319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:54.342720 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:54.342736 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:54.343310 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:54.343317 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:54.343572 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:54.344378 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:33:54.344435 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:33:54.346219 (+ 1784us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:33:54.348439 (+ 2220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:54.348449 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:54.348473 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:54.348998 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:54.349330 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:54.349336 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:54.349341 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:54.349587 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:54.350077 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:33:54.350085 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:33:54.350490 (+ 405us) client_negotiation.cc:769] Sending connection context
0810 22:33:54.350733 (+ 243us) client_negotiation.cc:240] Negotiation successful
0810 22:33:54.351161 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":132,"threads_started":1}
WARNING: no policy specified for kudu/127.16.233.67@KRBTEST.COM; defaulting to no policy
I20260810 22:33:54.354087 20390 heartbeater.cc:347] Connected to a master server at 127.16.233.126:38957
I20260810 22:33:54.354364 20390 heartbeater.cc:464] Registering TS with master...
I20260810 22:33:54.354894 20390 heartbeater.cc:511] Master 127.16.233.126:38957 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.67@KRBTEST.COM" created.
I20260810 22:33:54.356329 20037 ts_manager.cc:194] Registered new tserver with Master: 826d60aa26564e539d87037cca08c8b3 (127.16.233.66:41585)
I20260810 22:33:54.357035 20037 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.66@KRBTEST.COM'} at 127.16.233.66:52503
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab.
Entry for principal kudu/127.16.233.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab.
I20260810 22:33:54.376716 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.67:0
--local_ip_for_outbound_sockets=127.16.233.67
--webserver_interface=127.16.233.67
--webserver_port=0
--tserver_master_addrs=127.16.233.126:38957
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab
--principal=kudu/127.16.233.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:41351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:33:54.498342 20396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:33:54.498656 20396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:33:54.498751 20396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:33:54.503300 20396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:33:54.503391 20396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:33:54.503487 20396 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.67
I20260810 22:33:54.509253 20396 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:41351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab
--principal=kudu/127.16.233.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.233.67
--webserver_port=0
--tserver_master_addrs=127.16.233.126:38957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20396
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.67
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:33:54.510506 20396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:33:54.511633 20396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:33:54.523089 20403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:33:54.523084 20405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:33:54.523084 20402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:33:54.523476 20396 server_base.cc:1034] running on GCE node
I20260810 22:33:54.524121 20396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:33:54.524997 20396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:33:54.526232 20396 hybrid_clock.cc:648] HybridClock initialized: now 1786401234526214 us; error 37 us; skew 500 ppm
Aug 10 22:33:54 dist-test-slave-77v9 krb5kdc[19983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401234, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:33:54.529973 20396 init.cc:377] Logged in from keytab as kudu/127.16.233.67@KRBTEST.COM (short username kudu)
I20260810 22:33:54.531113 20396 webserver.cc:460] Webserver started at http://127.16.233.67:46677/ using document root <none> and password file <none>
I20260810 22:33:54.531780 20396 fs_manager.cc:359] Metadata directory not provided
I20260810 22:33:54.531893 20396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:33:54.532163 20396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:33:54.534729 20396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-2/data/instance:
uuid: "0c6155aca66b4055af5f5865f0dabe61"
format_stamp: "Formatted at 2026-08-10 22:33:54 on dist-test-slave-77v9"
server_key: "8648364ece77e4e5e94f20d4d51471e6"
server_key_iv: "b225b2fe1b8feded1c31494d4e78a248"
server_key_version: "encryptionkey@0"
I20260810 22:33:54.535421 20396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-2/wal/instance:
uuid: "0c6155aca66b4055af5f5865f0dabe61"
format_stamp: "Formatted at 2026-08-10 22:33:54 on dist-test-slave-77v9"
server_key: "8648364ece77e4e5e94f20d4d51471e6"
server_key_iv: "b225b2fe1b8feded1c31494d4e78a248"
server_key_version: "encryptionkey@0"
I20260810 22:33:54.540299 20396 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260810 22:33:54.543228 20413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:54.544989 20396 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260810 22:33:54.545176 20396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-2/wal
uuid: "0c6155aca66b4055af5f5865f0dabe61"
format_stamp: "Formatted at 2026-08-10 22:33:54 on dist-test-slave-77v9"
server_key: "8648364ece77e4e5e94f20d4d51471e6"
server_key_iv: "b225b2fe1b8feded1c31494d4e78a248"
server_key_version: "encryptionkey@0"
I20260810 22:33:54.545339 20396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:33:54.558668 20396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:33:54.562232 20396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:33:54.562500 20396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:33:54.563223 20396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:33:54.564178 20396 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:33:54.564260 20396 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:54.564334 20396 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:33:54.564401 20396 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:54.575215 20396 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.67:34021
I20260810 22:33:54.575302 20534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.67:34021 every 8 connection(s)
I20260810 22:33:54.576350 20396 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
I20260810 22:33:54.585444 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 20396
I20260810 22:33:54.585601 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-2/wal/instance
I20260810 22:33:54.585963 17317 external_mini_cluster.cc:1428] Setting key ac621c64e45dcecfc3650afeff3e5bcc
Aug 10 22:33:54 dist-test-slave-77v9 krb5kdc[19983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401234, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.67@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:33:54.591993 20080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:54.578530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:40257 (local address 127.16.233.126:38957)
0810 22:33:54.578693 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:54.578697 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:54.579803 (+ 1106us) server_negotiation.cc:407] Connection header received
0810 22:33:54.580754 (+ 951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:54.580757 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:54.580862 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:54.580955 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:54.582302 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:54.582879 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:54.584147 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:54.584378 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:54.584471 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:54.588076 (+ 3605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:54.588079 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:54.588081 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:54.588110 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:54.589827 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:54.590419 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:54.590422 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:54.590424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:54.590513 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:54.590891 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:54.590896 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:54.590897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:54.591108 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:54.591195 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:54.591741 (+ 546us) server_negotiation.cc:299] Negotiation successful
0810 22:33:54.591855 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 22:33:54.593047 20538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:54.578873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:38957 (local address 127.16.233.67:40257)
0810 22:33:54.579617 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:54.579666 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:54.580510 (+ 844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:54.581163 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:54.581170 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:54.581573 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:33:54.582135 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:54.582150 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:54.583011 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:54.583014 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:54.584002 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:54.584012 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:54.584540 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:54.584543 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:54.584706 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:54.585649 (+ 943us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:33:54.585673 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:33:54.587878 (+ 2205us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:33:54.589999 (+ 2121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:54.590005 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:54.590021 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:54.590317 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:54.590600 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:54.590603 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:54.590604 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:54.590767 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:54.591208 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:33:54.591213 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:33:54.591511 (+ 298us) client_negotiation.cc:769] Sending connection context
0810 22:33:54.591685 (+ 174us) client_negotiation.cc:240] Negotiation successful
0810 22:33:54.591997 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":136,"threads_started":1}
I20260810 22:33:54.594305 20536 heartbeater.cc:347] Connected to a master server at 127.16.233.126:38957
I20260810 22:33:54.594575 20536 heartbeater.cc:464] Registering TS with master...
I20260810 22:33:54.595139 20536 heartbeater.cc:511] Master 127.16.233.126:38957 requested a full tablet report, sending...
I20260810 22:33:54.596320 20037 ts_manager.cc:194] Registered new tserver with Master: 0c6155aca66b4055af5f5865f0dabe61 (127.16.233.67:34021)
I20260810 22:33:54.597069 20037 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.67@KRBTEST.COM'} at 127.16.233.67:40257
I20260810 22:33:54.600781 17317 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 22:33:54.618033 20080 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:33:54.616034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47506 (local address 127.16.233.126:38957)
0810 22:33:54.616189 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:54.616193 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:54.616398 (+ 205us) server_negotiation.cc:407] Connection header received
0810 22:33:54.616590 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:54.616594 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:54.616659 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:54.616753 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:54.617831 (+ 1078us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47506: BlockingRecv error: recv got EOF from 127.0.0.1:47506 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 22:33:54.619212 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 20105
I20260810 22:33:54.627704 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 20248
I20260810 22:33:54.639696 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 20396
I20260810 22:33:54.649719 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 19998
2026-08-10T22:33:54Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2430 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786401215402979-17317-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:33:54 dist-test-slave-77v9 krb5kdc[20552](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:33:54 dist-test-slave-77v9 krb5kdc[20552](info): set up 2 sockets
Aug 10 22:33:54 dist-test-slave-77v9 krb5kdc[20552](info): commencing operation
krb5kdc: starting...
W20260810 22:33:55.833155 17317 mini_kdc.cc:121] Time spent starting KDC: real 1.156s user 0.002s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Aug 10 22:33:55 dist-test-slave-77v9 krb5kdc[20552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:33:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:33:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.233.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab.
Entry for principal kudu/127.16.233.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab.
I20260810 22:33:55.981060 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786401215402979-17317-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786401215402979-17317-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786401215402979-17317-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:34413
--webserver_interface=127.16.233.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab
--principal=kudu/127.16.233.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:35575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.126:34413
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:33:56.103216 20567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:33:56.103523 20567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:33:56.103565 20567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:33:56.107811 20567 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:33:56.107904 20567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:33:56.107924 20567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:33:56.107942 20567 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:33:56.107957 20567 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:33:56.113686 20567 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:35575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786401215402979-17317-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786401215402979-17317-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.126:34413
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab
--principal=kudu/127.16.233.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:34413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786401215402979-17317-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:33:56.114850 20567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:33:56.115913 20567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:33:56.125465 20572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:33:56.125572 20567 server_base.cc:1034] running on GCE node
W20260810 22:33:56.125473 20573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:33:56.125463 20575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:33:56.126434 20567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:33:56.127887 20567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:33:56.129098 20567 hybrid_clock.cc:648] HybridClock initialized: now 1786401236129063 us; error 53 us; skew 500 ppm
Aug 10 22:33:56 dist-test-slave-77v9 krb5kdc[20552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401236, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:33:56.132508 20567 init.cc:377] Logged in from keytab as kudu/127.16.233.126@KRBTEST.COM (short username kudu)
I20260810 22:33:56.133612 20567 webserver.cc:460] Webserver started at http://127.16.233.126:37527/ using document root <none> and password file <none>
I20260810 22:33:56.134347 20567 fs_manager.cc:359] Metadata directory not provided
I20260810 22:33:56.134467 20567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:33:56.134757 20567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:33:56.137259 20567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786401215402979-17317-0/minicluster-data/master-0/data/instance:
uuid: "13b0e283967441479f7515bbedc6017f"
format_stamp: "Formatted at 2026-08-10 22:33:56 on dist-test-slave-77v9"
server_key: "6cb9c5fea0655bff396d1eaaed05fa2a"
server_key_iv: "c68075b2bef1dd66da34523710668f5c"
server_key_version: "encryptionkey@0"
I20260810 22:33:56.137974 20567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786401215402979-17317-0/minicluster-data/master-0/wal/instance:
uuid: "13b0e283967441479f7515bbedc6017f"
format_stamp: "Formatted at 2026-08-10 22:33:56 on dist-test-slave-77v9"
server_key: "6cb9c5fea0655bff396d1eaaed05fa2a"
server_key_iv: "c68075b2bef1dd66da34523710668f5c"
server_key_version: "encryptionkey@0"
I20260810 22:33:56.142619 20567 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 22:33:56.145514 20582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:56.146961 20567 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:33:56.147141 20567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786401215402979-17317-0/minicluster-data/master-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786401215402979-17317-0/minicluster-data/master-0/wal
uuid: "13b0e283967441479f7515bbedc6017f"
format_stamp: "Formatted at 2026-08-10 22:33:56 on dist-test-slave-77v9"
server_key: "6cb9c5fea0655bff396d1eaaed05fa2a"
server_key_iv: "c68075b2bef1dd66da34523710668f5c"
server_key_version: "encryptionkey@0"
I20260810 22:33:56.147286 20567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786401215402979-17317-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786401215402979-17317-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786401215402979-17317-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:33:56.171170 20567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:33:56.175020 20567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:33:56.175266 20567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:33:56.182919 20567 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.126:34413
I20260810 22:33:56.183348 20642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.126:34413 every 8 connection(s)
I20260810 22:33:56.184032 20567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
I20260810 22:33:56.187458 20644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:33:56.190193 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 20567
I20260810 22:33:56.190339 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786401215402979-17317-0/minicluster-data/master-0/wal/instance
I20260810 22:33:56.190646 17317 external_mini_cluster.cc:1428] Setting key 4693efd48a4f71d513473480c72fd000
I20260810 22:33:56.194612 20644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13b0e283967441479f7515bbedc6017f: Bootstrap starting.
I20260810 22:33:56.197657 20644 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13b0e283967441479f7515bbedc6017f: Neither blocks nor log segments found. Creating new log.
I20260810 22:33:56.198596 20644 log.cc:824] T 00000000000000000000000000000000 P 13b0e283967441479f7515bbedc6017f: Log is configured to *not* fsync() on all Append() calls
Aug 10 22:33:56 dist-test-slave-77v9 krb5kdc[20552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:33:56.201776 20644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13b0e283967441479f7515bbedc6017f: No bootstrap required, opened a new log
I20260810 22:33:56.207412 20644 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13b0e283967441479f7515bbedc6017f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13b0e283967441479f7515bbedc6017f" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 34413 } }
I20260810 22:33:56.207737 20644 raft_consensus.cc:374] T 00000000000000000000000000000000 P 13b0e283967441479f7515bbedc6017f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:33:56.207798 20644 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13b0e283967441479f7515bbedc6017f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13b0e283967441479f7515bbedc6017f, State: Initialized, Role: FOLLOWER
I20260810 22:33:56.207803 20647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:56.192092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50936 (local address 127.16.233.126:34413)
0810 22:33:56.192557 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:56.192573 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:56.192611 (+ 38us) server_negotiation.cc:407] Connection header received
0810 22:33:56.193376 (+ 765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:56.193405 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:56.193755 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:56.194065 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:56.195214 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:56.196229 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:56.197683 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:56.198297 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:56.198431 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:56.201659 (+ 3228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:56.201666 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:56.201685 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:56.201721 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:56.203952 (+ 2231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:56.204480 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:56.204490 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:56.204498 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:56.204701 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:56.205126 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:56.205131 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:56.205136 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:56.205728 (+ 592us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:56.206001 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:56.206270 (+ 269us) server_negotiation.cc:299] Negotiation successful
0810 22:33:56.206659 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":153,"threads_started":1}
I20260810 22:33:56.208402 20644 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13b0e283967441479f7515bbedc6017f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13b0e283967441479f7515bbedc6017f" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 34413 } }
I20260810 22:33:56.208600 20644 raft_consensus.cc:388] T 00000000000000000000000000000000 P 13b0e283967441479f7515bbedc6017f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:33:56.208707 20644 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13b0e283967441479f7515bbedc6017f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:33:56.208886 20644 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13b0e283967441479f7515bbedc6017f [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:33:56.210162 20644 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13b0e283967441479f7515bbedc6017f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13b0e283967441479f7515bbedc6017f" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 34413 } }
I20260810 22:33:56.210610 20644 leader_election.cc:302] T 00000000000000000000000000000000 P 13b0e283967441479f7515bbedc6017f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13b0e283967441479f7515bbedc6017f; no voters:
I20260810 22:33:56.211038 20644 leader_election.cc:288] T 00000000000000000000000000000000 P 13b0e283967441479f7515bbedc6017f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:33:56.211129 20649 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13b0e283967441479f7515bbedc6017f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:33:56.211192 17317 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 22:33:56.211386 20649 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13b0e283967441479f7515bbedc6017f [term 1 LEADER]: Becoming Leader. State: Replica: 13b0e283967441479f7515bbedc6017f, State: Running, Role: LEADER
I20260810 22:33:56.211793 20649 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13b0e283967441479f7515bbedc6017f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13b0e283967441479f7515bbedc6017f" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 34413 } }
I20260810 22:33:56.212534 20644 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13b0e283967441479f7515bbedc6017f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:33:56.213176 20649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13b0e283967441479f7515bbedc6017f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13b0e283967441479f7515bbedc6017f. Latest consensus state: current_term: 1 leader_uuid: "13b0e283967441479f7515bbedc6017f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13b0e283967441479f7515bbedc6017f" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 34413 } } }
I20260810 22:33:56.214118 20650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13b0e283967441479f7515bbedc6017f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "13b0e283967441479f7515bbedc6017f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13b0e283967441479f7515bbedc6017f" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 34413 } } }
I20260810 22:33:56.214262 20649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13b0e283967441479f7515bbedc6017f [sys.catalog]: This master's current role is: LEADER
I20260810 22:33:56.214354 20650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13b0e283967441479f7515bbedc6017f [sys.catalog]: This master's current role is: LEADER
I20260810 22:33:56.215214 20660 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:33:56.218268 20660 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:33:56.225440 20660 catalog_manager.cc:1347] Generated new cluster ID: fb9340db8cfa46acb3b3d44848489009
I20260810 22:33:56.225522 20660 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:33:56.235642 20660 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:33:56.237123 20660 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:33:56.239447 20647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:56.233802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50938 (local address 127.16.233.126:34413)
0810 22:33:56.233950 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:56.233957 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:56.234240 (+ 283us) server_negotiation.cc:407] Connection header received
0810 22:33:56.234390 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:56.234393 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:56.234452 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:56.234554 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:56.235305 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:56.236007 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:56.236938 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:56.237213 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:56.237318 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:56.238423 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:56.238427 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:56.238430 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:33:56.238928 (+ 498us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:56.238960 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:33:56.238976 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:56.239083 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:56.239193 (+ 110us) server_negotiation.cc:299] Negotiation successful
0810 22:33:56.239284 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 22:33:56.246243 20660 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 13b0e283967441479f7515bbedc6017f: Generated new TSK 0
I20260810 22:33:56.247119 20660 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:33:56.258973 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 20567
2026-08-10T22:33:56Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1607 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:33:56 dist-test-slave-77v9 krb5kdc[20678](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:33:56 dist-test-slave-77v9 krb5kdc[20678](info): set up 2 sockets
Aug 10 22:33:56 dist-test-slave-77v9 krb5kdc[20678](info): commencing operation
krb5kdc: starting...
W20260810 22:33:57.452464 17317 mini_kdc.cc:121] Time spent starting KDC: real 1.169s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Aug 10 22:33:57 dist-test-slave-77v9 krb5kdc[20678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:33:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:33:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.233.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab.
Entry for principal kudu/127.16.233.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab.
I20260810 22:33:57.601470 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:46241
--webserver_interface=127.16.233.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab
--principal=kudu/127.16.233.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:40281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.126:46241
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:33:57.730293 20694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:33:57.730652 20694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:33:57.730741 20694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:33:57.735422 20694 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:33:57.735548 20694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:33:57.735584 20694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:33:57.735626 20694 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:33:57.735661 20694 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:33:57.741714 20694 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:40281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.126:46241
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab
--principal=kudu/127.16.233.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:46241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:33:57.742971 20694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:33:57.744076 20694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:33:57.753952 20703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:33:57.753933 20700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:33:57.753957 20701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:33:57.754354 20694 server_base.cc:1034] running on GCE node
I20260810 22:33:57.754978 20694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:33:57.756349 20694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:33:57.757593 20694 hybrid_clock.cc:648] HybridClock initialized: now 1786401237757562 us; error 47 us; skew 500 ppm
Aug 10 22:33:57 dist-test-slave-77v9 krb5kdc[20678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401237, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:33:57.766321 20694 init.cc:377] Logged in from keytab as kudu/127.16.233.126@KRBTEST.COM (short username kudu)
I20260810 22:33:57.767468 20694 webserver.cc:460] Webserver started at http://127.16.233.126:40409/ using document root <none> and password file <none>
I20260810 22:33:57.768131 20694 fs_manager.cc:359] Metadata directory not provided
I20260810 22:33:57.768245 20694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:33:57.768507 20694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:33:57.770885 20694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/minicluster-data/master-0/data/instance:
uuid: "a078b3b43f4a4646a3d0d71b35ab92b7"
format_stamp: "Formatted at 2026-08-10 22:33:57 on dist-test-slave-77v9"
server_key: "2892967df9e8bef552e36a1fdb37b4a9"
server_key_iv: "804c1d11d152036ca8b3616ed196a42a"
server_key_version: "encryptionkey@0"
I20260810 22:33:57.771543 20694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/minicluster-data/master-0/wal/instance:
uuid: "a078b3b43f4a4646a3d0d71b35ab92b7"
format_stamp: "Formatted at 2026-08-10 22:33:57 on dist-test-slave-77v9"
server_key: "2892967df9e8bef552e36a1fdb37b4a9"
server_key_iv: "804c1d11d152036ca8b3616ed196a42a"
server_key_version: "encryptionkey@0"
I20260810 22:33:57.775909 20694 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 22:33:57.779109 20712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:57.780684 20694 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 22:33:57.780918 20694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/minicluster-data/master-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/minicluster-data/master-0/wal
uuid: "a078b3b43f4a4646a3d0d71b35ab92b7"
format_stamp: "Formatted at 2026-08-10 22:33:57 on dist-test-slave-77v9"
server_key: "2892967df9e8bef552e36a1fdb37b4a9"
server_key_iv: "804c1d11d152036ca8b3616ed196a42a"
server_key_version: "encryptionkey@0"
I20260810 22:33:57.781083 20694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:33:57.791321 20694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:33:57.794925 20694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:33:57.795198 20694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:33:57.802983 20694 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.126:46241
I20260810 22:33:57.803287 20765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.126:46241 every 8 connection(s)
I20260810 22:33:57.804097 20694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
I20260810 22:33:57.807082 20766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:33:57.810349 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 20694
I20260810 22:33:57.810518 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/minicluster-data/master-0/wal/instance
I20260810 22:33:57.810802 17317 external_mini_cluster.cc:1428] Setting key 02b8bc57d3c294df78c94035f11d9e83
I20260810 22:33:57.813694 20766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a078b3b43f4a4646a3d0d71b35ab92b7: Bootstrap starting.
I20260810 22:33:57.816463 20766 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a078b3b43f4a4646a3d0d71b35ab92b7: Neither blocks nor log segments found. Creating new log.
I20260810 22:33:57.817519 20766 log.cc:824] T 00000000000000000000000000000000 P a078b3b43f4a4646a3d0d71b35ab92b7: Log is configured to *not* fsync() on all Append() calls
I20260810 22:33:57.819907 20766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a078b3b43f4a4646a3d0d71b35ab92b7: No bootstrap required, opened a new log
Aug 10 22:33:57 dist-test-slave-77v9 krb5kdc[20678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:33:57.825109 20766 raft_consensus.cc:348] T 00000000000000000000000000000000 P a078b3b43f4a4646a3d0d71b35ab92b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a078b3b43f4a4646a3d0d71b35ab92b7" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 46241 } }
I20260810 22:33:57.825474 20766 raft_consensus.cc:374] T 00000000000000000000000000000000 P a078b3b43f4a4646a3d0d71b35ab92b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:33:57.825579 20766 raft_consensus.cc:729] T 00000000000000000000000000000000 P a078b3b43f4a4646a3d0d71b35ab92b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a078b3b43f4a4646a3d0d71b35ab92b7, State: Initialized, Role: FOLLOWER
I20260810 22:33:57.826102 20766 consensus_queue.cc:260] T 00000000000000000000000000000000 P a078b3b43f4a4646a3d0d71b35ab92b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a078b3b43f4a4646a3d0d71b35ab92b7" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 46241 } }
I20260810 22:33:57.826272 20766 raft_consensus.cc:388] T 00000000000000000000000000000000 P a078b3b43f4a4646a3d0d71b35ab92b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:33:57.826359 20766 raft_consensus.cc:482] T 00000000000000000000000000000000 P a078b3b43f4a4646a3d0d71b35ab92b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:33:57.826488 20766 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a078b3b43f4a4646a3d0d71b35ab92b7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:33:57.827016 20770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:57.812325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45748 (local address 127.16.233.126:46241)
0810 22:33:57.812848 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:57.812877 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:57.812927 (+ 50us) server_negotiation.cc:407] Connection header received
0810 22:33:57.813586 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:57.813604 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:57.814010 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:57.814321 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:57.815349 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:57.816391 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:57.817704 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:57.818258 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:57.818430 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:57.821147 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:57.821156 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:57.821174 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:57.821207 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:57.823421 (+ 2214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:57.823916 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:57.823923 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:57.823937 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:57.824098 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:57.824388 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:57.824392 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:57.824395 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:57.825009 (+ 614us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:57.825220 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:57.825532 (+ 312us) server_negotiation.cc:299] Negotiation successful
0810 22:33:57.825878 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":186,"threads_started":1}
I20260810 22:33:57.827859 20766 raft_consensus.cc:504] T 00000000000000000000000000000000 P a078b3b43f4a4646a3d0d71b35ab92b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a078b3b43f4a4646a3d0d71b35ab92b7" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 46241 } }
I20260810 22:33:57.828249 20766 leader_election.cc:302] T 00000000000000000000000000000000 P a078b3b43f4a4646a3d0d71b35ab92b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a078b3b43f4a4646a3d0d71b35ab92b7; no voters:
I20260810 22:33:57.828619 20766 leader_election.cc:288] T 00000000000000000000000000000000 P a078b3b43f4a4646a3d0d71b35ab92b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:33:57.828771 20772 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a078b3b43f4a4646a3d0d71b35ab92b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:33:57.829087 20772 raft_consensus.cc:686] T 00000000000000000000000000000000 P a078b3b43f4a4646a3d0d71b35ab92b7 [term 1 LEADER]: Becoming Leader. State: Replica: a078b3b43f4a4646a3d0d71b35ab92b7, State: Running, Role: LEADER
I20260810 22:33:57.829490 20772 consensus_queue.cc:237] T 00000000000000000000000000000000 P a078b3b43f4a4646a3d0d71b35ab92b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a078b3b43f4a4646a3d0d71b35ab92b7" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 46241 } }
I20260810 22:33:57.830093 20766 sys_catalog.cc:564] T 00000000000000000000000000000000 P a078b3b43f4a4646a3d0d71b35ab92b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:33:57.831383 20773 sys_catalog.cc:455] T 00000000000000000000000000000000 P a078b3b43f4a4646a3d0d71b35ab92b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a078b3b43f4a4646a3d0d71b35ab92b7. Latest consensus state: current_term: 1 leader_uuid: "a078b3b43f4a4646a3d0d71b35ab92b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a078b3b43f4a4646a3d0d71b35ab92b7" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 46241 } } }
I20260810 22:33:57.831537 20773 sys_catalog.cc:458] T 00000000000000000000000000000000 P a078b3b43f4a4646a3d0d71b35ab92b7 [sys.catalog]: This master's current role is: LEADER
I20260810 22:33:57.831715 20774 sys_catalog.cc:455] T 00000000000000000000000000000000 P a078b3b43f4a4646a3d0d71b35ab92b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a078b3b43f4a4646a3d0d71b35ab92b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a078b3b43f4a4646a3d0d71b35ab92b7" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 46241 } } }
I20260810 22:33:57.831902 20774 sys_catalog.cc:458] T 00000000000000000000000000000000 P a078b3b43f4a4646a3d0d71b35ab92b7 [sys.catalog]: This master's current role is: LEADER
I20260810 22:33:57.832219 20785 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:33:57.835264 20785 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:33:57.842176 20785 catalog_manager.cc:1347] Generated new cluster ID: 57ebfe65ac69438aadf44a3984da2d27
I20260810 22:33:57.842361 20785 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:33:57.868207 20785 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:33:57.869506 20785 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:33:57.879393 20785 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a078b3b43f4a4646a3d0d71b35ab92b7: Generated new TSK 0
I20260810 22:33:57.880389 20785 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.233.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab.
Entry for principal kudu/127.16.233.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab.
I20260810 22:33:57.925493 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.65:0
--local_ip_for_outbound_sockets=127.16.233.65
--webserver_interface=127.16.233.65
--webserver_port=0
--tserver_master_addrs=127.16.233.126:46241
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab
--principal=kudu/127.16.233.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:40281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:33:58.061168 20795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:33:58.061467 20795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:33:58.061513 20795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:33:58.065558 20795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:33:58.065622 20795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:33:58.065682 20795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.65
I20260810 22:33:58.071173 20795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:40281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab
--principal=kudu/127.16.233.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.233.65
--webserver_port=0
--tserver_master_addrs=127.16.233.126:46241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20795
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.65
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:33:58.072312 20795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:33:58.073375 20795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:33:58.084477 20801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:33:58.084486 20802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:33:58.084429 20804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:33:58.084451 20795 server_base.cc:1034] running on GCE node
I20260810 22:33:58.085289 20795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:33:58.086179 20795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:33:58.087417 20795 hybrid_clock.cc:648] HybridClock initialized: now 1786401238087376 us; error 56 us; skew 500 ppm
Aug 10 22:33:58 dist-test-slave-77v9 krb5kdc[20678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401238, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:33:58.090778 20795 init.cc:377] Logged in from keytab as kudu/127.16.233.65@KRBTEST.COM (short username kudu)
I20260810 22:33:58.091894 20795 webserver.cc:460] Webserver started at http://127.16.233.65:42375/ using document root <none> and password file <none>
I20260810 22:33:58.092577 20795 fs_manager.cc:359] Metadata directory not provided
I20260810 22:33:58.092686 20795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:33:58.093065 20795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:33:58.095544 20795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-0/data/instance:
uuid: "4799514ef51e46b49ba96687957e0d5a"
format_stamp: "Formatted at 2026-08-10 22:33:58 on dist-test-slave-77v9"
server_key: "595e51c253b76fc43f2d1204232c2582"
server_key_iv: "c1253a033b65786d30c0294ee8767482"
server_key_version: "encryptionkey@0"
I20260810 22:33:58.096271 20795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-0/wal/instance:
uuid: "4799514ef51e46b49ba96687957e0d5a"
format_stamp: "Formatted at 2026-08-10 22:33:58 on dist-test-slave-77v9"
server_key: "595e51c253b76fc43f2d1204232c2582"
server_key_iv: "c1253a033b65786d30c0294ee8767482"
server_key_version: "encryptionkey@0"
I20260810 22:33:58.101114 20795 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260810 22:33:58.104506 20811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:58.106391 20795 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:33:58.106603 20795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-0/wal
uuid: "4799514ef51e46b49ba96687957e0d5a"
format_stamp: "Formatted at 2026-08-10 22:33:58 on dist-test-slave-77v9"
server_key: "595e51c253b76fc43f2d1204232c2582"
server_key_iv: "c1253a033b65786d30c0294ee8767482"
server_key_version: "encryptionkey@0"
I20260810 22:33:58.106769 20795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:33:58.128984 20795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:33:58.133473 20795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:33:58.133724 20795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:33:58.134466 20795 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:33:58.135453 20795 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:33:58.135541 20795 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:58.135623 20795 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:33:58.135681 20795 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:58.145743 20795 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.65:44665
I20260810 22:33:58.145816 20935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.65:44665 every 8 connection(s)
I20260810 22:33:58.146837 20795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
I20260810 22:33:58.155949 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 20795
I20260810 22:33:58.156129 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-0/wal/instance
I20260810 22:33:58.156457 17317 external_mini_cluster.cc:1428] Setting key 73747be8799d45ee1507382e09060fa8
Aug 10 22:33:58 dist-test-slave-77v9 krb5kdc[20678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401238, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.65@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:33:58.163878 20770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:58.149002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:43713 (local address 127.16.233.126:46241)
0810 22:33:58.149187 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:58.149192 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:58.150311 (+ 1119us) server_negotiation.cc:407] Connection header received
0810 22:33:58.151843 (+ 1532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:58.151846 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:58.151896 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:58.151985 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:58.153450 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:58.154212 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:58.155525 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:58.156007 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:58.156149 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:58.159246 (+ 3097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:58.159253 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:58.159259 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:58.159297 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:58.161149 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:58.161986 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:58.161990 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:58.161991 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:58.162081 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:58.162562 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:58.162565 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:58.162566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:58.162763 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:58.162841 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:58.163587 (+ 746us) server_negotiation.cc:299] Negotiation successful
0810 22:33:58.163737 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 22:33:58.165387 20938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:58.149504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:46241 (local address 127.16.233.65:43713)
0810 22:33:58.150127 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:58.150186 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:58.151611 (+ 1425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:58.152171 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:58.152178 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:58.152594 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:33:58.153280 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:58.153295 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:58.154361 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:58.154364 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:58.155364 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:58.155374 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:58.156240 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:58.156245 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:58.156444 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:58.157298 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:33:58.157332 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:33:58.159056 (+ 1724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:33:58.161284 (+ 2228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:58.161292 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:58.161316 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:58.161773 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:58.162198 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:58.162205 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:58.162211 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:58.162443 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:58.162879 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:33:58.162890 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:33:58.163296 (+ 406us) client_negotiation.cc:769] Sending connection context
0810 22:33:58.163531 (+ 235us) client_negotiation.cc:240] Negotiation successful
0810 22:33:58.164006 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":159,"threads_started":1}
I20260810 22:33:58.167055 20936 heartbeater.cc:347] Connected to a master server at 127.16.233.126:46241
I20260810 22:33:58.167361 20936 heartbeater.cc:464] Registering TS with master...
I20260810 22:33:58.167994 20936 heartbeater.cc:511] Master 127.16.233.126:46241 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.16.233.66@KRBTEST.COM; defaulting to no policy
I20260810 22:33:58.169823 20730 ts_manager.cc:194] Registered new tserver with Master: 4799514ef51e46b49ba96687957e0d5a (127.16.233.65:44665)
I20260810 22:33:58.171283 20730 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.65@KRBTEST.COM'} at 127.16.233.65:43713
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab.
Entry for principal kudu/127.16.233.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab.
I20260810 22:33:58.192669 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.66:0
--local_ip_for_outbound_sockets=127.16.233.66
--webserver_interface=127.16.233.66
--webserver_port=0
--tserver_master_addrs=127.16.233.126:46241
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab
--principal=kudu/127.16.233.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:40281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:33:58.309911 20942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:33:58.310220 20942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:33:58.310269 20942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:33:58.314679 20942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:33:58.314764 20942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:33:58.314833 20942 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.66
I20260810 22:33:58.320619 20942 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:40281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab
--principal=kudu/127.16.233.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.233.66
--webserver_port=0
--tserver_master_addrs=127.16.233.126:46241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20942
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.66
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:33:58.321914 20942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:33:58.322955 20942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:33:58.333952 20947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:33:58.333978 20950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:33:58.333976 20948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:33:58.334360 20942 server_base.cc:1034] running on GCE node
I20260810 22:33:58.334908 20942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:33:58.335615 20942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:33:58.336900 20942 hybrid_clock.cc:648] HybridClock initialized: now 1786401238336870 us; error 30 us; skew 500 ppm
Aug 10 22:33:58 dist-test-slave-77v9 krb5kdc[20678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401238, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:33:58.340050 20942 init.cc:377] Logged in from keytab as kudu/127.16.233.66@KRBTEST.COM (short username kudu)
I20260810 22:33:58.341107 20942 webserver.cc:460] Webserver started at http://127.16.233.66:42277/ using document root <none> and password file <none>
I20260810 22:33:58.341638 20942 fs_manager.cc:359] Metadata directory not provided
I20260810 22:33:58.341718 20942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:33:58.341893 20942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:33:58.343945 20942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-1/data/instance:
uuid: "71291eb69d2d4b7196d21f96c2d1e804"
format_stamp: "Formatted at 2026-08-10 22:33:58 on dist-test-slave-77v9"
server_key: "9bd9c971e58a7305637b40b2814176ba"
server_key_iv: "2748027d125d1af9cef3a8594b7a9e71"
server_key_version: "encryptionkey@0"
I20260810 22:33:58.344480 20942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-1/wal/instance:
uuid: "71291eb69d2d4b7196d21f96c2d1e804"
format_stamp: "Formatted at 2026-08-10 22:33:58 on dist-test-slave-77v9"
server_key: "9bd9c971e58a7305637b40b2814176ba"
server_key_iv: "2748027d125d1af9cef3a8594b7a9e71"
server_key_version: "encryptionkey@0"
I20260810 22:33:58.348424 20942 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 22:33:58.351197 20957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:58.352643 20942 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 22:33:58.352756 20942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-1/wal
uuid: "71291eb69d2d4b7196d21f96c2d1e804"
format_stamp: "Formatted at 2026-08-10 22:33:58 on dist-test-slave-77v9"
server_key: "9bd9c971e58a7305637b40b2814176ba"
server_key_iv: "2748027d125d1af9cef3a8594b7a9e71"
server_key_version: "encryptionkey@0"
I20260810 22:33:58.352928 20942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:33:58.374450 20942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:33:58.378293 20942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:33:58.378475 20942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:33:58.379195 20942 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:33:58.380198 20942 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:33:58.380290 20942 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:58.380366 20942 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:33:58.380419 20942 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:58.390707 20942 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.66:46847
I20260810 22:33:58.390878 21079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.66:46847 every 8 connection(s)
I20260810 22:33:58.391839 20942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
I20260810 22:33:58.401667 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 20942
I20260810 22:33:58.401808 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-1/wal/instance
I20260810 22:33:58.402096 17317 external_mini_cluster.cc:1428] Setting key b1f3e35bcfa0592f49516a98ab6b5c90
Aug 10 22:33:58 dist-test-slave-77v9 krb5kdc[20678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401238, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.66@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:33:58.408284 20770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:58.394221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:38383 (local address 127.16.233.126:46241)
0810 22:33:58.394384 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:58.394388 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:58.395344 (+ 956us) server_negotiation.cc:407] Connection header received
0810 22:33:58.396716 (+ 1372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:58.396720 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:58.396774 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:58.396938 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:58.398350 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:58.399014 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:58.400347 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:58.400604 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:58.400707 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:58.403944 (+ 3237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:58.403947 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:58.403950 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:58.403975 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:58.405495 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:58.406257 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:58.406264 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:58.406267 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:58.406388 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:58.406776 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:58.406782 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:58.406784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:58.407088 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:58.407253 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:58.407821 (+ 568us) server_negotiation.cc:299] Negotiation successful
0810 22:33:58.408004 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 22:33:58.409219 21082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:58.394634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:46241 (local address 127.16.233.66:38383)
0810 22:33:58.395158 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:58.395207 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:58.396472 (+ 1265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:58.397149 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:58.397157 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:58.397578 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:33:58.398119 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:58.398134 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:58.399207 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:58.399211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:58.400186 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:58.400197 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:58.400770 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:58.400772 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:58.401054 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:58.401796 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:33:58.401823 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:33:58.403750 (+ 1927us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:33:58.405629 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:58.405639 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:58.405661 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:58.406099 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:58.406518 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:58.406520 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:58.406521 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:58.406668 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:58.407250 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:33:58.407256 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:33:58.407540 (+ 284us) client_negotiation.cc:769] Sending connection context
0810 22:33:58.407722 (+ 182us) client_negotiation.cc:240] Negotiation successful
0810 22:33:58.408097 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":124,"threads_started":1}
I20260810 22:33:58.410634 21080 heartbeater.cc:347] Connected to a master server at 127.16.233.126:46241
I20260810 22:33:58.410952 21080 heartbeater.cc:464] Registering TS with master...
I20260810 22:33:58.411700 21080 heartbeater.cc:511] Master 127.16.233.126:46241 requested a full tablet report, sending...
I20260810 22:33:58.413136 20730 ts_manager.cc:194] Registered new tserver with Master: 71291eb69d2d4b7196d21f96c2d1e804 (127.16.233.66:46847)
I20260810 22:33:58.413769 20730 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.66@KRBTEST.COM'} at 127.16.233.66:38383
WARNING: no policy specified for kudu/127.16.233.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab.
Entry for principal kudu/127.16.233.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab.
I20260810 22:33:58.437089 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.67:0
--local_ip_for_outbound_sockets=127.16.233.67
--webserver_interface=127.16.233.67
--webserver_port=0
--tserver_master_addrs=127.16.233.126:46241
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab
--principal=kudu/127.16.233.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:40281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:33:58.558248 21085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:33:58.558573 21085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:33:58.558621 21085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:33:58.563120 21085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:33:58.563208 21085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:33:58.563272 21085 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.67
I20260810 22:33:58.568996 21085 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:40281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab
--principal=kudu/127.16.233.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.233.67
--webserver_port=0
--tserver_master_addrs=127.16.233.126:46241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21085
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.67
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:33:58.570180 21085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:33:58.571250 21085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:33:58.582693 21091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:33:58.582962 21085 server_base.cc:1034] running on GCE node
W20260810 22:33:58.582721 21090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:33:58.582746 21094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:33:58.583712 21085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:33:58.584546 21085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:33:58.585772 21085 hybrid_clock.cc:648] HybridClock initialized: now 1786401238585745 us; error 57 us; skew 500 ppm
Aug 10 22:33:58 dist-test-slave-77v9 krb5kdc[20678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401238, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:33:58.589208 21085 init.cc:377] Logged in from keytab as kudu/127.16.233.67@KRBTEST.COM (short username kudu)
I20260810 22:33:58.590329 21085 webserver.cc:460] Webserver started at http://127.16.233.67:34071/ using document root <none> and password file <none>
I20260810 22:33:58.590975 21085 fs_manager.cc:359] Metadata directory not provided
I20260810 22:33:58.591048 21085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:33:58.591298 21085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:33:58.593757 21085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-2/data/instance:
uuid: "cf0e3633e8604877bd287b5baa502615"
format_stamp: "Formatted at 2026-08-10 22:33:58 on dist-test-slave-77v9"
server_key: "a32ee549f32a53c24a1a28118f2b7f15"
server_key_iv: "734cd08337346a56e0423e4b1e5c6679"
server_key_version: "encryptionkey@0"
I20260810 22:33:58.594436 21085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-2/wal/instance:
uuid: "cf0e3633e8604877bd287b5baa502615"
format_stamp: "Formatted at 2026-08-10 22:33:58 on dist-test-slave-77v9"
server_key: "a32ee549f32a53c24a1a28118f2b7f15"
server_key_iv: "734cd08337346a56e0423e4b1e5c6679"
server_key_version: "encryptionkey@0"
I20260810 22:33:58.599030 21085 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 22:33:58.602053 21101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:58.603503 21085 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 22:33:58.603653 21085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-2/wal
uuid: "cf0e3633e8604877bd287b5baa502615"
format_stamp: "Formatted at 2026-08-10 22:33:58 on dist-test-slave-77v9"
server_key: "a32ee549f32a53c24a1a28118f2b7f15"
server_key_iv: "734cd08337346a56e0423e4b1e5c6679"
server_key_version: "encryptionkey@0"
I20260810 22:33:58.603837 21085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:33:58.617913 21085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:33:58.621510 21085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:33:58.621766 21085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:33:58.622470 21085 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:33:58.623445 21085 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:33:58.623528 21085 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:58.623607 21085 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:33:58.623667 21085 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:58.634796 21085 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.67:35401
I20260810 22:33:58.634821 21222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.67:35401 every 8 connection(s)
I20260810 22:33:58.635890 21085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
I20260810 22:33:58.645875 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 21085
I20260810 22:33:58.646030 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786401215402979-17317-0/minicluster-data/ts-2/wal/instance
I20260810 22:33:58.646342 17317 external_mini_cluster.cc:1428] Setting key 8904cf63d90079e86030023ba501553f
Aug 10 22:33:58 dist-test-slave-77v9 krb5kdc[20678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401238, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.67@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:33:58.652014 20770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:58.638202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:46681 (local address 127.16.233.126:46241)
0810 22:33:58.638393 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:58.638399 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:58.639436 (+ 1037us) server_negotiation.cc:407] Connection header received
0810 22:33:58.640953 (+ 1517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:58.640956 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:58.641012 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:58.641104 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:58.642434 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:58.642997 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:58.644288 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:58.644560 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:58.644671 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:58.647982 (+ 3311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:58.647985 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:58.647988 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:58.648042 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:58.649712 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:58.650298 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:58.650302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:58.650303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:58.650401 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:58.650791 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:58.650795 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:58.650797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:58.651032 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:58.651150 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:58.651683 (+ 533us) server_negotiation.cc:299] Negotiation successful
0810 22:33:58.651843 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 22:33:58.653004 21225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:58.638722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:46241 (local address 127.16.233.67:46681)
0810 22:33:58.639255 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:58.639301 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:58.640620 (+ 1319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:58.641296 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:58.641303 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:58.641698 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:33:58.642249 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:58.642263 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:58.643147 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:58.643157 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:58.644115 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:58.644125 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:58.644721 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:58.644723 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:58.644935 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:58.645669 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:33:58.645694 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:33:58.647765 (+ 2071us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:33:58.649853 (+ 2088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:58.649861 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:58.649886 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:58.650179 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:58.650511 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:58.650513 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:58.650516 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:58.650687 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:58.651150 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:33:58.651158 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:33:58.651440 (+ 282us) client_negotiation.cc:769] Sending connection context
0810 22:33:58.651641 (+ 201us) client_negotiation.cc:240] Negotiation successful
0810 22:33:58.652015 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":144,"threads_started":1}
I20260810 22:33:58.654274 21223 heartbeater.cc:347] Connected to a master server at 127.16.233.126:46241
I20260810 22:33:58.654558 21223 heartbeater.cc:464] Registering TS with master...
I20260810 22:33:58.655277 21223 heartbeater.cc:511] Master 127.16.233.126:46241 requested a full tablet report, sending...
I20260810 22:33:58.656535 20730 ts_manager.cc:194] Registered new tserver with Master: cf0e3633e8604877bd287b5baa502615 (127.16.233.67:35401)
I20260810 22:33:58.657337 20730 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.67@KRBTEST.COM'} at 127.16.233.67:46681
I20260810 22:33:58.661168 17317 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 22:33:58 dist-test-slave-77v9 krb5kdc[20678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401238, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 10 22:33:58 dist-test-slave-77v9 krb5kdc[20678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401238, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:33:58.691303 20770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:58.681268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45756 (local address 127.16.233.126:46241)
0810 22:33:58.681429 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:58.681434 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:58.681604 (+ 170us) server_negotiation.cc:407] Connection header received
0810 22:33:58.681710 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:58.681712 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:58.681756 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:58.681839 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:58.682660 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:58.683327 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:58.684521 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:58.684880 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:58.685014 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:58.688007 (+ 2993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:58.688011 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:58.688013 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:58.688045 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:58.689569 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:58.690071 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:58.690074 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:58.690075 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:58.690162 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:58.690442 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:58.690444 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:58.690446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:58.690703 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:58.690810 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:58.691059 (+ 249us) server_negotiation.cc:299] Negotiation successful
0810 22:33:58.691174 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260810 22:33:58.691885 20730 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:45756
I20260810 22:33:58.693310 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 20795
I20260810 22:33:58.702309 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 20942
I20260810 22:33:58.710305 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 21085
I20260810 22:33:58.718712 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 20694
2026-08-10T22:33:58Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2462 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:33:58 dist-test-slave-77v9 krb5kdc[21236](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:33:58 dist-test-slave-77v9 krb5kdc[21236](info): set up 2 sockets
Aug 10 22:33:58 dist-test-slave-77v9 krb5kdc[21236](info): commencing operation
krb5kdc: starting...
W20260810 22:33:59.895785 17317 mini_kdc.cc:121] Time spent starting KDC: real 1.149s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Aug 10 22:33:59 dist-test-slave-77v9 krb5kdc[21236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:34:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:34:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.233.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab.
Entry for principal kudu/127.16.233.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab.
I20260810 22:34:00.045096 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:35381
--webserver_interface=127.16.233.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab
--principal=kudu/127.16.233.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:44043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.126:35381
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:34:00.167585 21250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:34:00.168025 21250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:34:00.168135 21250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:34:00.172384 21250 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:34:00.172498 21250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:34:00.172535 21250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:34:00.172572 21250 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:34:00.172605 21250 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:34:00.178398 21250 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:44043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.126:35381
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab
--principal=kudu/127.16.233.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:35381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:34:00.179677 21250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:34:00.180858 21250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:34:00.190231 21255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:00.190371 21257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:00.190465 21250 server_base.cc:1034] running on GCE node
W20260810 22:34:00.190380 21259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:00.191215 21250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:34:00.192538 21250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:34:00.193774 21250 hybrid_clock.cc:648] HybridClock initialized: now 1786401240193758 us; error 69 us; skew 500 ppm
Aug 10 22:34:00 dist-test-slave-77v9 krb5kdc[21236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401240, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:34:00.197355 21250 init.cc:377] Logged in from keytab as kudu/127.16.233.126@KRBTEST.COM (short username kudu)
I20260810 22:34:00.198532 21250 webserver.cc:460] Webserver started at http://127.16.233.126:39173/ using document root <none> and password file <none>
I20260810 22:34:00.199200 21250 fs_manager.cc:359] Metadata directory not provided
I20260810 22:34:00.199311 21250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:34:00.199579 21250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:34:00.202178 21250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/minicluster-data/master-0/data/instance:
uuid: "8a68dc05e05141a8ac0f9e45f2a4179b"
format_stamp: "Formatted at 2026-08-10 22:34:00 on dist-test-slave-77v9"
server_key: "3bc313dcf2649278febc9d02be0b13ce"
server_key_iv: "959a67f6de557046069125e40b73f8e0"
server_key_version: "encryptionkey@0"
I20260810 22:34:00.202852 21250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/minicluster-data/master-0/wal/instance:
uuid: "8a68dc05e05141a8ac0f9e45f2a4179b"
format_stamp: "Formatted at 2026-08-10 22:34:00 on dist-test-slave-77v9"
server_key: "3bc313dcf2649278febc9d02be0b13ce"
server_key_iv: "959a67f6de557046069125e40b73f8e0"
server_key_version: "encryptionkey@0"
I20260810 22:34:00.207607 21250 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 22:34:00.210647 21267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:00.212109 21250 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 22:34:00.212283 21250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/minicluster-data/master-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/minicluster-data/master-0/wal
uuid: "8a68dc05e05141a8ac0f9e45f2a4179b"
format_stamp: "Formatted at 2026-08-10 22:34:00 on dist-test-slave-77v9"
server_key: "3bc313dcf2649278febc9d02be0b13ce"
server_key_iv: "959a67f6de557046069125e40b73f8e0"
server_key_version: "encryptionkey@0"
I20260810 22:34:00.212419 21250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:34:00.230854 21250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:34:00.234367 21250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:34:00.234598 21250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:34:00.243191 21250 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.126:35381
I20260810 22:34:00.243218 21323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.126:35381 every 8 connection(s)
I20260810 22:34:00.244297 21250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
I20260810 22:34:00.247517 21324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:34:00.253750 21324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a68dc05e05141a8ac0f9e45f2a4179b: Bootstrap starting.
I20260810 22:34:00.254441 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 21250
I20260810 22:34:00.254598 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/minicluster-data/master-0/wal/instance
I20260810 22:34:00.254961 17317 external_mini_cluster.cc:1428] Setting key 11e939f6d84eb852d496b728942139e4
I20260810 22:34:00.256801 21324 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a68dc05e05141a8ac0f9e45f2a4179b: Neither blocks nor log segments found. Creating new log.
I20260810 22:34:00.257705 21324 log.cc:824] T 00000000000000000000000000000000 P 8a68dc05e05141a8ac0f9e45f2a4179b: Log is configured to *not* fsync() on all Append() calls
I20260810 22:34:00.259873 21324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a68dc05e05141a8ac0f9e45f2a4179b: No bootstrap required, opened a new log
I20260810 22:34:00.263862 21324 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a68dc05e05141a8ac0f9e45f2a4179b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a68dc05e05141a8ac0f9e45f2a4179b" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 35381 } }
I20260810 22:34:00.264142 21324 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a68dc05e05141a8ac0f9e45f2a4179b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:34:00.264227 21324 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a68dc05e05141a8ac0f9e45f2a4179b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a68dc05e05141a8ac0f9e45f2a4179b, State: Initialized, Role: FOLLOWER
Aug 10 22:34:00 dist-test-slave-77v9 krb5kdc[21236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:34:00.264755 21324 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a68dc05e05141a8ac0f9e45f2a4179b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a68dc05e05141a8ac0f9e45f2a4179b" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 35381 } }
I20260810 22:34:00.264962 21324 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a68dc05e05141a8ac0f9e45f2a4179b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:34:00.265048 21324 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a68dc05e05141a8ac0f9e45f2a4179b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:34:00.265151 21324 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a68dc05e05141a8ac0f9e45f2a4179b [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:34:00.266582 21324 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a68dc05e05141a8ac0f9e45f2a4179b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a68dc05e05141a8ac0f9e45f2a4179b" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 35381 } }
I20260810 22:34:00.267019 21324 leader_election.cc:302] T 00000000000000000000000000000000 P 8a68dc05e05141a8ac0f9e45f2a4179b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a68dc05e05141a8ac0f9e45f2a4179b; no voters:
I20260810 22:34:00.267410 21324 leader_election.cc:288] T 00000000000000000000000000000000 P 8a68dc05e05141a8ac0f9e45f2a4179b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:34:00.267575 21329 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a68dc05e05141a8ac0f9e45f2a4179b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:34:00.267853 21329 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a68dc05e05141a8ac0f9e45f2a4179b [term 1 LEADER]: Becoming Leader. State: Replica: 8a68dc05e05141a8ac0f9e45f2a4179b, State: Running, Role: LEADER
I20260810 22:34:00.268268 21329 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a68dc05e05141a8ac0f9e45f2a4179b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a68dc05e05141a8ac0f9e45f2a4179b" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 35381 } }
I20260810 22:34:00.268992 21324 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a68dc05e05141a8ac0f9e45f2a4179b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:34:00.270124 21331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a68dc05e05141a8ac0f9e45f2a4179b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a68dc05e05141a8ac0f9e45f2a4179b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a68dc05e05141a8ac0f9e45f2a4179b" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 35381 } } }
I20260810 22:34:00.270262 21331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a68dc05e05141a8ac0f9e45f2a4179b [sys.catalog]: This master's current role is: LEADER
I20260810 22:34:00.270514 21331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a68dc05e05141a8ac0f9e45f2a4179b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a68dc05e05141a8ac0f9e45f2a4179b. Latest consensus state: current_term: 1 leader_uuid: "8a68dc05e05141a8ac0f9e45f2a4179b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a68dc05e05141a8ac0f9e45f2a4179b" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 35381 } } }
I20260810 22:34:00.270596 21331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a68dc05e05141a8ac0f9e45f2a4179b [sys.catalog]: This master's current role is: LEADER
I20260810 22:34:00.272018 21327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:00.256244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44320 (local address 127.16.233.126:35381)
0810 22:34:00.256724 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:00.256745 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:00.256793 (+ 48us) server_negotiation.cc:407] Connection header received
0810 22:34:00.257619 (+ 826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:00.257637 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:00.258046 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:00.258411 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:00.259334 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:00.260412 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:00.261368 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:00.261873 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:00.262091 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:00.265275 (+ 3184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:00.265283 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:00.265299 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:00.265351 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:00.267598 (+ 2247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:00.268286 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:00.268292 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:00.268298 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:00.268460 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:00.268825 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:00.268829 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:00.268831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:00.269422 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:00.269633 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:00.270532 (+ 899us) server_negotiation.cc:299] Negotiation successful
0810 22:34:00.270906 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":138,"threads_started":1}
W20260810 22:34:00.274677 21344 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8a68dc05e05141a8ac0f9e45f2a4179b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 22:34:00.274773 21344 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 22:34:00.274850 21338 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:34:00.275563 21338 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:34:00.281507 21338 catalog_manager.cc:1347] Generated new cluster ID: 14e0286c53604a838e62790b2177fbbf
I20260810 22:34:00.281613 21338 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:34:00.294690 21338 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:34:00.296098 21338 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:34:00.305481 21338 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a68dc05e05141a8ac0f9e45f2a4179b: Generated new TSK 0
I20260810 22:34:00.306396 21338 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.233.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab.
Entry for principal kudu/127.16.233.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab.
I20260810 22:34:00.348547 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.65:0
--local_ip_for_outbound_sockets=127.16.233.65
--webserver_interface=127.16.233.65
--webserver_port=0
--tserver_master_addrs=127.16.233.126:35381
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab
--principal=kudu/127.16.233.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:44043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:34:00.491168 21351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:34:00.491511 21351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:34:00.491560 21351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:34:00.495985 21351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:34:00.496089 21351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:34:00.496222 21351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.65
I20260810 22:34:00.502112 21351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:44043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab
--principal=kudu/127.16.233.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.233.65
--webserver_port=0
--tserver_master_addrs=127.16.233.126:35381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.65
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:34:00.503392 21351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:34:00.504580 21351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:34:00.516777 21358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:00.516798 21357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:00.516983 21351 server_base.cc:1034] running on GCE node
W20260810 22:34:00.516834 21360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:00.517638 21351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:34:00.518392 21351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:34:00.519572 21351 hybrid_clock.cc:648] HybridClock initialized: now 1786401240519537 us; error 53 us; skew 500 ppm
Aug 10 22:34:00 dist-test-slave-77v9 krb5kdc[21236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401240, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:34:00.522684 21351 init.cc:377] Logged in from keytab as kudu/127.16.233.65@KRBTEST.COM (short username kudu)
I20260810 22:34:00.523679 21351 webserver.cc:460] Webserver started at http://127.16.233.65:38789/ using document root <none> and password file <none>
I20260810 22:34:00.524264 21351 fs_manager.cc:359] Metadata directory not provided
I20260810 22:34:00.524331 21351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:34:00.524516 21351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:34:00.526863 21351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/minicluster-data/ts-0/data/instance:
uuid: "4055ebc2b67340cbbdb11e8f4824651b"
format_stamp: "Formatted at 2026-08-10 22:34:00 on dist-test-slave-77v9"
server_key: "d62c256199e2945eabac002ae2820494"
server_key_iv: "292c73d31626b1976efe5ac05499ecaa"
server_key_version: "encryptionkey@0"
I20260810 22:34:00.527482 21351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/minicluster-data/ts-0/wal/instance:
uuid: "4055ebc2b67340cbbdb11e8f4824651b"
format_stamp: "Formatted at 2026-08-10 22:34:00 on dist-test-slave-77v9"
server_key: "d62c256199e2945eabac002ae2820494"
server_key_iv: "292c73d31626b1976efe5ac05499ecaa"
server_key_version: "encryptionkey@0"
I20260810 22:34:00.531764 21351 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 22:34:00.534761 21368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:00.536381 21351 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 22:34:00.536514 21351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/minicluster-data/ts-0/wal
uuid: "4055ebc2b67340cbbdb11e8f4824651b"
format_stamp: "Formatted at 2026-08-10 22:34:00 on dist-test-slave-77v9"
server_key: "d62c256199e2945eabac002ae2820494"
server_key_iv: "292c73d31626b1976efe5ac05499ecaa"
server_key_version: "encryptionkey@0"
I20260810 22:34:00.536628 21351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:34:00.547654 21351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:34:00.551433 21351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:34:00.551748 21351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:34:00.552443 21351 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:34:00.553439 21351 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:34:00.553514 21351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:00.553586 21351 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:34:00.553637 21351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:00.563256 21351 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.65:36019
I20260810 22:34:00.563299 21496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.65:36019 every 8 connection(s)
I20260810 22:34:00.564323 21351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
I20260810 22:34:00.569154 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 21351
I20260810 22:34:00.569272 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/minicluster-data/ts-0/wal/instance
I20260810 22:34:00.569715 17317 external_mini_cluster.cc:1428] Setting key fc060f4bb3c8be7481862a00c8a82ebe
Aug 10 22:34:00 dist-test-slave-77v9 krb5kdc[21236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401240, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.65@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.233.66@KRBTEST.COM; defaulting to no policy
I20260810 22:34:00.582237 21327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:00.566602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:42375 (local address 127.16.233.126:35381)
0810 22:34:00.566769 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:00.566773 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:00.568161 (+ 1388us) server_negotiation.cc:407] Connection header received
0810 22:34:00.569557 (+ 1396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:00.569561 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:00.569636 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:00.569715 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:00.571301 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:00.571894 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:00.573297 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:00.573558 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:00.573688 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:00.577497 (+ 3809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:00.577502 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:00.577504 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:00.577536 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:00.579204 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:00.580020 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:00.580023 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:00.580024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:00.580114 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:00.580671 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:00.580674 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:00.580675 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:00.580994 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:00.581091 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:00.581937 (+ 846us) server_negotiation.cc:299] Negotiation successful
0810 22:34:00.582100 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 22:34:00.583288 21499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:00.567105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:35381 (local address 127.16.233.65:42375)
0810 22:34:00.567943 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:00.568018 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:00.569265 (+ 1247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:00.569915 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:00.569922 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:00.570550 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:00.571117 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:00.571132 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:00.572071 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:00.572074 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:00.573125 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:00.573137 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:00.573762 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:00.573767 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:00.574028 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:00.574836 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:00.574864 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:00.577197 (+ 2333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:34:00.579381 (+ 2184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:00.579392 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:00.579415 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:00.579875 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:00.580219 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:00.580233 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:00.580237 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:00.580463 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:00.581124 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:34:00.581134 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:34:00.581585 (+ 451us) client_negotiation.cc:769] Sending connection context
0810 22:34:00.581814 (+ 229us) client_negotiation.cc:240] Negotiation successful
0810 22:34:00.582274 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":129,"threads_started":1}
I20260810 22:34:00.584813 21497 heartbeater.cc:347] Connected to a master server at 127.16.233.126:35381
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.66@KRBTEST.COM" created.
I20260810 22:34:00.585211 21497 heartbeater.cc:464] Registering TS with master...
I20260810 22:34:00.585780 21497 heartbeater.cc:511] Master 127.16.233.126:35381 requested a full tablet report, sending...
I20260810 22:34:00.587533 21284 ts_manager.cc:194] Registered new tserver with Master: 4055ebc2b67340cbbdb11e8f4824651b (127.16.233.65:36019)
I20260810 22:34:00.588770 21284 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.65@KRBTEST.COM'} at 127.16.233.65:42375
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab.
Entry for principal kudu/127.16.233.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab.
I20260810 22:34:00.605090 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.66:0
--local_ip_for_outbound_sockets=127.16.233.66
--webserver_interface=127.16.233.66
--webserver_port=0
--tserver_master_addrs=127.16.233.126:35381
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab
--principal=kudu/127.16.233.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:44043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:34:00.726131 21502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:34:00.726493 21502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:34:00.726545 21502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:34:00.730965 21502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:34:00.731063 21502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:34:00.731127 21502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.66
I20260810 22:34:00.737591 21502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:44043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab
--principal=kudu/127.16.233.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.233.66
--webserver_port=0
--tserver_master_addrs=127.16.233.126:35381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21502
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.66
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:34:00.739409 21502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:34:00.740903 21502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:34:00.751842 21508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:00.751893 21510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:00.752080 21502 server_base.cc:1034] running on GCE node
W20260810 22:34:00.751910 21507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:00.752646 21502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:34:00.753502 21502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:34:00.754724 21502 hybrid_clock.cc:648] HybridClock initialized: now 1786401240754697 us; error 48 us; skew 500 ppm
Aug 10 22:34:00 dist-test-slave-77v9 krb5kdc[21236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401240, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:34:00.757861 21502 init.cc:377] Logged in from keytab as kudu/127.16.233.66@KRBTEST.COM (short username kudu)
I20260810 22:34:00.758935 21502 webserver.cc:460] Webserver started at http://127.16.233.66:35007/ using document root <none> and password file <none>
I20260810 22:34:00.759565 21502 fs_manager.cc:359] Metadata directory not provided
I20260810 22:34:00.759667 21502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:34:00.759949 21502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:34:00.762372 21502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/minicluster-data/ts-1/data/instance:
uuid: "73a3074c9aac470facf53d63f32a6218"
format_stamp: "Formatted at 2026-08-10 22:34:00 on dist-test-slave-77v9"
server_key: "6c02e74785a9426775b31a0395943825"
server_key_iv: "6218cc697d0ece90740036dcbb42c095"
server_key_version: "encryptionkey@0"
I20260810 22:34:00.762997 21502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/minicluster-data/ts-1/wal/instance:
uuid: "73a3074c9aac470facf53d63f32a6218"
format_stamp: "Formatted at 2026-08-10 22:34:00 on dist-test-slave-77v9"
server_key: "6c02e74785a9426775b31a0395943825"
server_key_iv: "6218cc697d0ece90740036dcbb42c095"
server_key_version: "encryptionkey@0"
I20260810 22:34:00.767164 21502 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 22:34:00.770170 21517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:00.771700 21502 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:34:00.771813 21502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/minicluster-data/ts-1/wal
uuid: "73a3074c9aac470facf53d63f32a6218"
format_stamp: "Formatted at 2026-08-10 22:34:00 on dist-test-slave-77v9"
server_key: "6c02e74785a9426775b31a0395943825"
server_key_iv: "6218cc697d0ece90740036dcbb42c095"
server_key_version: "encryptionkey@0"
I20260810 22:34:00.771947 21502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:34:00.784907 21502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:34:00.788692 21502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:34:00.789013 21502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:34:00.789731 21502 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:34:00.790665 21502 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:34:00.790715 21502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:00.790757 21502 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:34:00.790773 21502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:00.800962 21502 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.66:46093
I20260810 22:34:00.801093 21640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.66:46093 every 8 connection(s)
I20260810 22:34:00.802101 21502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
I20260810 22:34:00.804316 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 21502
I20260810 22:34:00.804430 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/minicluster-data/ts-1/wal/instance
I20260810 22:34:00.804780 17317 external_mini_cluster.cc:1428] Setting key 4628cd6daf83684d5f993029bfbe120f
Aug 10 22:34:00 dist-test-slave-77v9 krb5kdc[21236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401240, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.66@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.233.67@KRBTEST.COM; defaulting to no policy
I20260810 22:34:00.820391 21327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:00.804584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:50457 (local address 127.16.233.126:35381)
0810 22:34:00.804781 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:00.804787 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:00.805547 (+ 760us) server_negotiation.cc:407] Connection header received
0810 22:34:00.807329 (+ 1782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:00.807336 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:00.807412 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:00.807512 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:00.808943 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:00.809818 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:00.812048 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:00.812279 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:00.812411 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:00.815977 (+ 3566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:00.815981 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:00.815983 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:00.816014 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:00.817836 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:00.818464 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:00.818467 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:00.818469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:00.818583 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:00.819012 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:00.819020 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:00.819022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:00.819231 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:00.819345 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:00.819980 (+ 635us) server_negotiation.cc:299] Negotiation successful
0810 22:34:00.820127 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.67@KRBTEST.COM" created.
I20260810 22:34:00.821514 21643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:00.804773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:35381 (local address 127.16.233.66:50457)
0810 22:34:00.805368 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:00.805430 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:00.807045 (+ 1615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:00.807700 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:00.807708 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:00.808200 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:00.808724 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:00.808737 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:00.810003 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:00.810007 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:00.811881 (+ 1874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:00.811900 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:00.812487 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:00.812496 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:00.812799 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:00.813567 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:00.813607 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:00.815741 (+ 2134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:34:00.817990 (+ 2249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:00.817997 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:00.818017 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:00.818322 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:00.818686 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:00.818689 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:00.818691 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:00.818895 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:00.819346 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:34:00.819355 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:34:00.819648 (+ 293us) client_negotiation.cc:769] Sending connection context
0810 22:34:00.819884 (+ 236us) client_negotiation.cc:240] Negotiation successful
0810 22:34:00.820322 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":160,"threads_started":1}
I20260810 22:34:00.823019 21641 heartbeater.cc:347] Connected to a master server at 127.16.233.126:35381
I20260810 22:34:00.823374 21641 heartbeater.cc:464] Registering TS with master...
I20260810 22:34:00.824097 21641 heartbeater.cc:511] Master 127.16.233.126:35381 requested a full tablet report, sending...
I20260810 22:34:00.825534 21284 ts_manager.cc:194] Registered new tserver with Master: 73a3074c9aac470facf53d63f32a6218 (127.16.233.66:46093)
I20260810 22:34:00.826220 21284 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.66@KRBTEST.COM'} at 127.16.233.66:50457
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab.
Entry for principal kudu/127.16.233.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab.
I20260810 22:34:00.841499 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.67:0
--local_ip_for_outbound_sockets=127.16.233.67
--webserver_interface=127.16.233.67
--webserver_port=0
--tserver_master_addrs=127.16.233.126:35381
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab
--principal=kudu/127.16.233.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:44043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:34:00.974902 21646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:34:00.975323 21646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:34:00.975389 21646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:34:00.980238 21646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:34:00.980334 21646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:34:00.980401 21646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.67
I20260810 22:34:00.986399 21646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:44043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab
--principal=kudu/127.16.233.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.233.67
--webserver_port=0
--tserver_master_addrs=127.16.233.126:35381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.67
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:34:00.987831 21646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:34:00.989053 21646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 22:34:00.999917 21646 server_base.cc:1034] running on GCE node
W20260810 22:34:00.999878 21654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:00.999799 21651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:00.999797 21652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:01.000754 21646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:34:01.001601 21646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:34:01.002872 21646 hybrid_clock.cc:648] HybridClock initialized: now 1786401241002855 us; error 34 us; skew 500 ppm
Aug 10 22:34:01 dist-test-slave-77v9 krb5kdc[21236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401241, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:34:01.006317 21646 init.cc:377] Logged in from keytab as kudu/127.16.233.67@KRBTEST.COM (short username kudu)
I20260810 22:34:01.007433 21646 webserver.cc:460] Webserver started at http://127.16.233.67:45339/ using document root <none> and password file <none>
I20260810 22:34:01.008162 21646 fs_manager.cc:359] Metadata directory not provided
I20260810 22:34:01.008277 21646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:34:01.008548 21646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:34:01.010959 21646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/minicluster-data/ts-2/data/instance:
uuid: "91ea34ef06e14f3bb3515283b498b83d"
format_stamp: "Formatted at 2026-08-10 22:34:01 on dist-test-slave-77v9"
server_key: "46e13f32d7a712aa0edda8a468faa8ba"
server_key_iv: "08b494bb011dc7f619c7dcacedfbd527"
server_key_version: "encryptionkey@0"
I20260810 22:34:01.011618 21646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/minicluster-data/ts-2/wal/instance:
uuid: "91ea34ef06e14f3bb3515283b498b83d"
format_stamp: "Formatted at 2026-08-10 22:34:01 on dist-test-slave-77v9"
server_key: "46e13f32d7a712aa0edda8a468faa8ba"
server_key_iv: "08b494bb011dc7f619c7dcacedfbd527"
server_key_version: "encryptionkey@0"
I20260810 22:34:01.016343 21646 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 22:34:01.019703 21661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:01.021492 21646 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:34:01.021675 21646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/minicluster-data/ts-2/wal
uuid: "91ea34ef06e14f3bb3515283b498b83d"
format_stamp: "Formatted at 2026-08-10 22:34:01 on dist-test-slave-77v9"
server_key: "46e13f32d7a712aa0edda8a468faa8ba"
server_key_iv: "08b494bb011dc7f619c7dcacedfbd527"
server_key_version: "encryptionkey@0"
I20260810 22:34:01.021824 21646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:34:01.044627 21646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:34:01.048568 21646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:34:01.048852 21646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:34:01.049647 21646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:34:01.050606 21646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:34:01.050685 21646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:01.050748 21646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:34:01.050798 21646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:01.061022 21646 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.67:33687
I20260810 22:34:01.061060 21783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.67:33687 every 8 connection(s)
I20260810 22:34:01.062182 21646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
I20260810 22:34:01.071674 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 21646
I20260810 22:34:01.071897 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786401215402979-17317-0/minicluster-data/ts-2/wal/instance
I20260810 22:34:01.072220 17317 external_mini_cluster.cc:1428] Setting key 6ccb1518fd8d388024f7828e42d08290
Aug 10 22:34:01 dist-test-slave-77v9 krb5kdc[21236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401241, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.67@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:34:01.080000 21327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:01.064399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:57147 (local address 127.16.233.126:35381)
0810 22:34:01.064572 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:01.064578 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:01.065731 (+ 1153us) server_negotiation.cc:407] Connection header received
0810 22:34:01.067473 (+ 1742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:01.067476 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:01.067534 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:01.067621 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:01.069316 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:01.070030 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:01.071663 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:01.072011 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:01.072152 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:01.075674 (+ 3522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:01.075678 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:01.075680 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:01.075709 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:01.077592 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:01.078329 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:01.078332 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:01.078334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:01.078425 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:01.078835 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:01.078838 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:01.078839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:01.079046 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:01.079122 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:01.079700 (+ 578us) server_negotiation.cc:299] Negotiation successful
0810 22:34:01.079826 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 22:34:01.081259 21786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:01.064943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:35381 (local address 127.16.233.67:57147)
0810 22:34:01.065543 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:01.065600 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:01.067180 (+ 1580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:01.067893 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:01.067903 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:01.068366 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:01.069092 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:01.069124 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:01.070280 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:01.070287 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:01.071491 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:01.071505 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:01.072232 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:01.072237 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:01.072466 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:01.073368 (+ 902us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:01.073392 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:01.075441 (+ 2049us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:34:01.077820 (+ 2379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:01.077827 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:01.077843 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:01.078185 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:01.078547 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:01.078550 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:01.078552 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:01.078732 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:01.079147 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:34:01.079152 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:34:01.079452 (+ 300us) client_negotiation.cc:769] Sending connection context
0810 22:34:01.079636 (+ 184us) client_negotiation.cc:240] Negotiation successful
0810 22:34:01.080037 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":144,"threads_started":1}
I20260810 22:34:01.082757 21784 heartbeater.cc:347] Connected to a master server at 127.16.233.126:35381
I20260810 22:34:01.083101 21784 heartbeater.cc:464] Registering TS with master...
I20260810 22:34:01.083806 21784 heartbeater.cc:511] Master 127.16.233.126:35381 requested a full tablet report, sending...
I20260810 22:34:01.085160 21284 ts_manager.cc:194] Registered new tserver with Master: 91ea34ef06e14f3bb3515283b498b83d (127.16.233.67:33687)
I20260810 22:34:01.085857 21284 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.67@KRBTEST.COM'} at 127.16.233.67:57147
I20260810 22:34:01.087265 17317 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 22:34:01 dist-test-slave-77v9 krb5kdc[21236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401241, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 22:34:01 dist-test-slave-77v9 krb5kdc[21236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401241, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.65@KRBTEST.COM
I20260810 22:34:01.118671 21791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:01.106121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52658 (local address 127.16.233.65:36019)
0810 22:34:01.106564 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:01.106571 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:01.106598 (+ 27us) server_negotiation.cc:407] Connection header received
0810 22:34:01.106655 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:01.106660 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:01.106871 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:01.107014 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:01.107870 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:01.108871 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:01.109888 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:01.110392 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:01.110528 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:01.113677 (+ 3149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:01.113694 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:01.113711 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:01.113746 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:01.116040 (+ 2294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:01.116675 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:01.116681 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:01.116683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:01.116955 (+ 272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:01.117353 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:01.117357 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:01.117359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:01.117668 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:01.117932 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:01.118227 (+ 295us) server_negotiation.cc:299] Negotiation successful
0810 22:34:01.118476 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":121,"threads_started":1}
Aug 10 22:34:01 dist-test-slave-77v9 krb5kdc[21236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401241, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:34:01.132675 21327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:01.122507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44328 (local address 127.16.233.126:35381)
0810 22:34:01.122683 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:01.122690 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:01.122774 (+ 84us) server_negotiation.cc:407] Connection header received
0810 22:34:01.122940 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:01.122943 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:01.123000 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:01.123093 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:01.123885 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:01.124422 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:01.125375 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:01.125608 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:01.125726 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:01.129044 (+ 3318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:01.129048 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:01.129050 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:01.129084 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:01.130769 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:01.131253 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:01.131256 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:01.131257 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:01.131339 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:01.131659 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:01.131661 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:01.131662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:01.131897 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:01.132020 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:01.132387 (+ 367us) server_negotiation.cc:299] Negotiation successful
0810 22:34:01.132536 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260810 22:34:01.133323 21284 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44328
I20260810 22:34:01.134286 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 21351
I20260810 22:34:01.144003 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 21502
I20260810 22:34:01.153041 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 21646
I20260810 22:34:01.161595 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 21250
2026-08-10T22:34:01Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2442 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:34:01 dist-test-slave-77v9 krb5kdc[21796](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:34:01 dist-test-slave-77v9 krb5kdc[21796](info): set up 2 sockets
Aug 10 22:34:01 dist-test-slave-77v9 krb5kdc[21796](info): commencing operation
krb5kdc: starting...
W20260810 22:34:02.344218 17317 mini_kdc.cc:121] Time spent starting KDC: real 1.156s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Aug 10 22:34:02 dist-test-slave-77v9 krb5kdc[21796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:34:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:34:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.233.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab.
Entry for principal kudu/127.16.233.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab.
I20260810 22:34:02.497689 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:36183
--webserver_interface=127.16.233.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab
--principal=kudu/127.16.233.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:46771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.126:36183
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:34:02.619494 21810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:34:02.619838 21810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:34:02.619886 21810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:34:02.624199 21810 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:34:02.624289 21810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:34:02.624310 21810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:34:02.624328 21810 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:34:02.624343 21810 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:34:02.630209 21810 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:46771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.126:36183
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab
--principal=kudu/127.16.233.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:36183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:34:02.631373 21810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:34:02.632450 21810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:34:02.642007 21818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:02.642246 21810 server_base.cc:1034] running on GCE node
W20260810 22:34:02.642025 21816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:02.642021 21815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:02.643059 21810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:34:02.644291 21810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:34:02.645478 21810 hybrid_clock.cc:648] HybridClock initialized: now 1786401242645464 us; error 31 us; skew 500 ppm
Aug 10 22:34:02 dist-test-slave-77v9 krb5kdc[21796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401242, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:34:02.648518 21810 init.cc:377] Logged in from keytab as kudu/127.16.233.126@KRBTEST.COM (short username kudu)
I20260810 22:34:02.649376 21810 fs_manager.cc:359] Metadata directory not provided
I20260810 22:34:02.649447 21810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:34:02.649626 21810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:34:02.651885 21810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/minicluster-data/master-0/data/instance:
uuid: "f2c09e72f1d44f109a30ee9f151676ce"
format_stamp: "Formatted at 2026-08-10 22:34:02 on dist-test-slave-77v9"
server_key: "2075397e2c430d823c4e4f3db53e99f0"
server_key_iv: "c274dab86f2bedf2974d37d9d1dd33b4"
server_key_version: "encryptionkey@0"
I20260810 22:34:02.652498 21810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/minicluster-data/master-0/wal/instance:
uuid: "f2c09e72f1d44f109a30ee9f151676ce"
format_stamp: "Formatted at 2026-08-10 22:34:02 on dist-test-slave-77v9"
server_key: "2075397e2c430d823c4e4f3db53e99f0"
server_key_iv: "c274dab86f2bedf2974d37d9d1dd33b4"
server_key_version: "encryptionkey@0"
I20260810 22:34:02.657141 21810 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260810 22:34:02.659864 21825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:02.661391 21810 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:34:02.661521 21810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/minicluster-data/master-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/minicluster-data/master-0/wal
uuid: "f2c09e72f1d44f109a30ee9f151676ce"
format_stamp: "Formatted at 2026-08-10 22:34:02 on dist-test-slave-77v9"
server_key: "2075397e2c430d823c4e4f3db53e99f0"
server_key_iv: "c274dab86f2bedf2974d37d9d1dd33b4"
server_key_version: "encryptionkey@0"
I20260810 22:34:02.661629 21810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:34:02.678583 21810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:34:02.682405 21810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:34:02.682603 21810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:34:02.692085 21810 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.126:36183
I20260810 22:34:02.692504 21882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.126:36183 every 8 connection(s)
I20260810 22:34:02.692938 21810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
I20260810 22:34:02.696179 21883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:34:02.697285 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 21810
I20260810 22:34:02.697490 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/minicluster-data/master-0/wal/instance
I20260810 22:34:02.697809 17317 external_mini_cluster.cc:1428] Setting key 0a5f1354066927a8166465179f14b3da
I20260810 22:34:02.703975 21883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2c09e72f1d44f109a30ee9f151676ce: Bootstrap starting.
I20260810 22:34:02.707165 21883 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2c09e72f1d44f109a30ee9f151676ce: Neither blocks nor log segments found. Creating new log.
I20260810 22:34:02.708134 21883 log.cc:824] T 00000000000000000000000000000000 P f2c09e72f1d44f109a30ee9f151676ce: Log is configured to *not* fsync() on all Append() calls
Aug 10 22:34:02 dist-test-slave-77v9 krb5kdc[21796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:34:02.710705 21883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2c09e72f1d44f109a30ee9f151676ce: No bootstrap required, opened a new log
I20260810 22:34:02.715171 21886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:02.699586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54456 (local address 127.16.233.126:36183)
0810 22:34:02.700123 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:02.700143 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:02.700191 (+ 48us) server_negotiation.cc:407] Connection header received
0810 22:34:02.701120 (+ 929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:02.701149 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:02.701633 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:02.702060 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:02.702969 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:02.704433 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:02.705586 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:02.706320 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:02.706536 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:02.709734 (+ 3198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:02.709741 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:02.709757 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:02.709788 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:02.711762 (+ 1974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:02.712352 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:02.712359 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:02.712364 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:02.712533 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:02.712909 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:02.712912 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:02.712914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:02.713372 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:02.713566 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:02.713910 (+ 344us) server_negotiation.cc:299] Negotiation successful
0810 22:34:02.714285 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":155,"threads_started":1}
I20260810 22:34:02.715308 21883 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2c09e72f1d44f109a30ee9f151676ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2c09e72f1d44f109a30ee9f151676ce" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 36183 } }
I20260810 22:34:02.715632 21883 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2c09e72f1d44f109a30ee9f151676ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:34:02.715724 21883 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2c09e72f1d44f109a30ee9f151676ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2c09e72f1d44f109a30ee9f151676ce, State: Initialized, Role: FOLLOWER
I20260810 22:34:02.716269 21883 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2c09e72f1d44f109a30ee9f151676ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2c09e72f1d44f109a30ee9f151676ce" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 36183 } }
I20260810 22:34:02.716455 21883 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2c09e72f1d44f109a30ee9f151676ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:34:02.716532 21883 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2c09e72f1d44f109a30ee9f151676ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:34:02.716635 21883 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2c09e72f1d44f109a30ee9f151676ce [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:34:02.717958 21883 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2c09e72f1d44f109a30ee9f151676ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2c09e72f1d44f109a30ee9f151676ce" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 36183 } }
I20260810 22:34:02.718345 21883 leader_election.cc:302] T 00000000000000000000000000000000 P f2c09e72f1d44f109a30ee9f151676ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2c09e72f1d44f109a30ee9f151676ce; no voters:
I20260810 22:34:02.718693 21883 leader_election.cc:288] T 00000000000000000000000000000000 P f2c09e72f1d44f109a30ee9f151676ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:34:02.719022 21888 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2c09e72f1d44f109a30ee9f151676ce [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:34:02.719257 21888 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2c09e72f1d44f109a30ee9f151676ce [term 1 LEADER]: Becoming Leader. State: Replica: f2c09e72f1d44f109a30ee9f151676ce, State: Running, Role: LEADER
I20260810 22:34:02.719583 21888 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2c09e72f1d44f109a30ee9f151676ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2c09e72f1d44f109a30ee9f151676ce" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 36183 } }
I20260810 22:34:02.720007 21883 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2c09e72f1d44f109a30ee9f151676ce [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:34:02.721485 21889 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2c09e72f1d44f109a30ee9f151676ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2c09e72f1d44f109a30ee9f151676ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2c09e72f1d44f109a30ee9f151676ce" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 36183 } } }
I20260810 22:34:02.721467 21891 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2c09e72f1d44f109a30ee9f151676ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2c09e72f1d44f109a30ee9f151676ce. Latest consensus state: current_term: 1 leader_uuid: "f2c09e72f1d44f109a30ee9f151676ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2c09e72f1d44f109a30ee9f151676ce" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 36183 } } }
I20260810 22:34:02.721647 21889 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2c09e72f1d44f109a30ee9f151676ce [sys.catalog]: This master's current role is: LEADER
I20260810 22:34:02.721647 21891 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2c09e72f1d44f109a30ee9f151676ce [sys.catalog]: This master's current role is: LEADER
I20260810 22:34:02.722059 21902 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:34:02.725217 21902 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:34:02.731112 21902 catalog_manager.cc:1347] Generated new cluster ID: 25bef6fbc37b4ec592ce2c7fce8c27d9
I20260810 22:34:02.731210 21902 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:34:02.744539 21902 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:34:02.746021 21902 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:34:02.754307 21902 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2c09e72f1d44f109a30ee9f151676ce: Generated new TSK 0
I20260810 22:34:02.755120 21902 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.233.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab.
Entry for principal kudu/127.16.233.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab.
I20260810 22:34:02.793496 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.65:0
--local_ip_for_outbound_sockets=127.16.233.65
--webserver_interface=127.16.233.65
--webserver_port=0
--tserver_master_addrs=127.16.233.126:36183
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab
--principal=kudu/127.16.233.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:46771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:34:02.950452 21913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:34:02.950754 21913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:34:02.950802 21913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:34:02.955946 21913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:34:02.956041 21913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:34:02.956116 21913 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.65
I20260810 22:34:02.962893 21913 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:46771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab
--principal=kudu/127.16.233.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.233.65
--webserver_port=0
--tserver_master_addrs=127.16.233.126:36183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21913
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.65
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:34:02.964138 21913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:34:02.965405 21913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:34:02.976608 21920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:02.976668 21919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:02.976977 21922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:02.977290 21913 server_base.cc:1034] running on GCE node
I20260810 22:34:02.977818 21913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:34:02.978533 21913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:34:02.979750 21913 hybrid_clock.cc:648] HybridClock initialized: now 1786401242979736 us; error 34 us; skew 500 ppm
Aug 10 22:34:02 dist-test-slave-77v9 krb5kdc[21796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401242, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:34:02.983093 21913 init.cc:377] Logged in from keytab as kudu/127.16.233.65@KRBTEST.COM (short username kudu)
I20260810 22:34:02.983963 21913 fs_manager.cc:359] Metadata directory not provided
I20260810 22:34:02.984040 21913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:34:02.984226 21913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:34:02.986550 21913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/minicluster-data/ts-0/data/instance:
uuid: "8d97593349144b6ea5e7786351bf40fd"
format_stamp: "Formatted at 2026-08-10 22:34:02 on dist-test-slave-77v9"
server_key: "776c73701581fe5d89b91f3a707a4a29"
server_key_iv: "ca93325d21ec41fa9e42c7b466421874"
server_key_version: "encryptionkey@0"
I20260810 22:34:02.987214 21913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/minicluster-data/ts-0/wal/instance:
uuid: "8d97593349144b6ea5e7786351bf40fd"
format_stamp: "Formatted at 2026-08-10 22:34:02 on dist-test-slave-77v9"
server_key: "776c73701581fe5d89b91f3a707a4a29"
server_key_iv: "ca93325d21ec41fa9e42c7b466421874"
server_key_version: "encryptionkey@0"
I20260810 22:34:02.991447 21913 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 22:34:02.994418 21929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:02.996214 21913 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 22:34:02.996357 21913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/minicluster-data/ts-0/wal
uuid: "8d97593349144b6ea5e7786351bf40fd"
format_stamp: "Formatted at 2026-08-10 22:34:02 on dist-test-slave-77v9"
server_key: "776c73701581fe5d89b91f3a707a4a29"
server_key_iv: "ca93325d21ec41fa9e42c7b466421874"
server_key_version: "encryptionkey@0"
I20260810 22:34:02.996495 21913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:34:03.010313 21913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:34:03.014356 21913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:34:03.014546 21913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:34:03.015164 21913 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:34:03.016075 21913 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:34:03.016127 21913 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:03.016170 21913 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:34:03.016186 21913 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:03.027493 21913 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.65:42229
I20260810 22:34:03.027601 22052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.65:42229 every 8 connection(s)
I20260810 22:34:03.028292 21913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
I20260810 22:34:03.033624 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 21913
I20260810 22:34:03.033753 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/minicluster-data/ts-0/wal/instance
I20260810 22:34:03.034123 17317 external_mini_cluster.cc:1428] Setting key 5d46595a3fabd477a39335105a506003
Aug 10 22:34:03 dist-test-slave-77v9 krb5kdc[21796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401242, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.65@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:34:03.045346 21886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:03.030349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:55621 (local address 127.16.233.126:36183)
0810 22:34:03.030509 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:03.030513 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:03.031929 (+ 1416us) server_negotiation.cc:407] Connection header received
0810 22:34:03.033030 (+ 1101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:03.033034 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:03.033100 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:03.033226 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:03.035030 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:03.035719 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:03.037145 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:03.037424 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:03.037554 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:03.041121 (+ 3567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:03.041125 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:03.041127 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:03.041157 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:03.042826 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:03.043482 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:03.043486 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:03.043488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:03.043574 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:03.043994 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:03.043997 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:03.043999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:03.044209 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:03.044330 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:03.044953 (+ 623us) server_negotiation.cc:299] Negotiation successful
0810 22:34:03.045132 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
WARNING: no policy specified for kudu/127.16.233.66@KRBTEST.COM; defaulting to no policy
I20260810 22:34:03.046356 22055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:03.030759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:36183 (local address 127.16.233.65:55621)
0810 22:34:03.031728 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:03.031781 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:03.032712 (+ 931us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:03.033463 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:03.033471 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:03.034029 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:03.034648 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:03.034665 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:03.035883 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:03.035886 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:03.036993 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:03.037005 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:03.037628 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:03.037634 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:03.037835 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:03.038714 (+ 879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:03.038733 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:03.040894 (+ 2161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:34:03.042992 (+ 2098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:03.042999 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:03.043020 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:03.043345 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:03.043689 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:03.043692 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:03.043694 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:03.043878 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:03.044344 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:34:03.044354 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:34:03.044647 (+ 293us) client_negotiation.cc:769] Sending connection context
0810 22:34:03.044882 (+ 235us) client_negotiation.cc:240] Negotiation successful
0810 22:34:03.045266 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":128,"threads_started":1}
I20260810 22:34:03.048118 22053 heartbeater.cc:347] Connected to a master server at 127.16.233.126:36183
I20260810 22:34:03.048442 22053 heartbeater.cc:464] Registering TS with master...
I20260810 22:34:03.049149 22053 heartbeater.cc:511] Master 127.16.233.126:36183 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.66@KRBTEST.COM" created.
I20260810 22:34:03.051113 21845 ts_manager.cc:194] Registered new tserver with Master: 8d97593349144b6ea5e7786351bf40fd (127.16.233.65:42229)
I20260810 22:34:03.052557 21845 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.65@KRBTEST.COM'} at 127.16.233.65:55621
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab.
Entry for principal kudu/127.16.233.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab.
I20260810 22:34:03.070715 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.66:0
--local_ip_for_outbound_sockets=127.16.233.66
--webserver_interface=127.16.233.66
--webserver_port=0
--tserver_master_addrs=127.16.233.126:36183
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab
--principal=kudu/127.16.233.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:46771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:34:03.201694 22058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:34:03.202106 22058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:34:03.202160 22058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:34:03.206964 22058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:34:03.207063 22058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:34:03.207137 22058 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.66
I20260810 22:34:03.213258 22058 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:46771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab
--principal=kudu/127.16.233.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.233.66
--webserver_port=0
--tserver_master_addrs=127.16.233.126:36183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.66
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:34:03.214679 22058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:34:03.215780 22058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:34:03.228005 22063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:03.228097 22066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:03.228276 22058 server_base.cc:1034] running on GCE node
W20260810 22:34:03.228029 22064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:03.229148 22058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:34:03.230000 22058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:34:03.231338 22058 hybrid_clock.cc:648] HybridClock initialized: now 1786401243231323 us; error 38 us; skew 500 ppm
Aug 10 22:34:03 dist-test-slave-77v9 krb5kdc[21796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401243, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:34:03.234721 22058 init.cc:377] Logged in from keytab as kudu/127.16.233.66@KRBTEST.COM (short username kudu)
I20260810 22:34:03.235622 22058 fs_manager.cc:359] Metadata directory not provided
I20260810 22:34:03.235702 22058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:34:03.235894 22058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:34:03.238366 22058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/minicluster-data/ts-1/data/instance:
uuid: "ce82c66759814208a320f1142be5ba16"
format_stamp: "Formatted at 2026-08-10 22:34:03 on dist-test-slave-77v9"
server_key: "34403b1a3368fe5b7ee6805f7c0fea94"
server_key_iv: "6a4235c5aa9bae4526184e09d6b6ba33"
server_key_version: "encryptionkey@0"
I20260810 22:34:03.239071 22058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/minicluster-data/ts-1/wal/instance:
uuid: "ce82c66759814208a320f1142be5ba16"
format_stamp: "Formatted at 2026-08-10 22:34:03 on dist-test-slave-77v9"
server_key: "34403b1a3368fe5b7ee6805f7c0fea94"
server_key_iv: "6a4235c5aa9bae4526184e09d6b6ba33"
server_key_version: "encryptionkey@0"
I20260810 22:34:03.243654 22058 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 22:34:03.246696 22072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:03.248172 22058 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 22:34:03.248299 22058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/minicluster-data/ts-1/wal
uuid: "ce82c66759814208a320f1142be5ba16"
format_stamp: "Formatted at 2026-08-10 22:34:03 on dist-test-slave-77v9"
server_key: "34403b1a3368fe5b7ee6805f7c0fea94"
server_key_iv: "6a4235c5aa9bae4526184e09d6b6ba33"
server_key_version: "encryptionkey@0"
I20260810 22:34:03.248485 22058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:34:03.267452 22058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:34:03.271384 22058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:34:03.271651 22058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:34:03.272365 22058 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:34:03.273459 22058 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:34:03.273521 22058 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:03.273598 22058 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:34:03.273633 22058 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:03.284968 22058 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.66:40959
I20260810 22:34:03.284976 22197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.66:40959 every 8 connection(s)
I20260810 22:34:03.285797 22058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
I20260810 22:34:03.290009 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 22058
I20260810 22:34:03.290138 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/minicluster-data/ts-1/wal/instance
I20260810 22:34:03.290517 17317 external_mini_cluster.cc:1428] Setting key 1e6a11301942d47154ccaa755625c0be
Aug 10 22:34:03 dist-test-slave-77v9 krb5kdc[21796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401243, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.66@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.233.67@KRBTEST.COM; defaulting to no policy
I20260810 22:34:03.305099 21886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:03.287999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:34085 (local address 127.16.233.126:36183)
0810 22:34:03.288197 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:03.288201 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:03.289484 (+ 1283us) server_negotiation.cc:407] Connection header received
0810 22:34:03.291378 (+ 1894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:03.291381 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:03.291463 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:03.291592 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:03.293517 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:03.294174 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:03.295580 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:03.295852 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:03.295991 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:03.299944 (+ 3953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:03.299952 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:03.299959 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:03.300009 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:03.302177 (+ 2168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:03.302959 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:03.302967 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:03.302970 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:03.303093 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:03.303606 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:03.303613 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:03.303615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:03.303885 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:03.304007 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:03.304736 (+ 729us) server_negotiation.cc:299] Negotiation successful
0810 22:34:03.304958 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260810 22:34:03.306120 22202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:03.288465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:36183 (local address 127.16.233.66:34085)
0810 22:34:03.289267 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:03.289324 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:03.291085 (+ 1761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:03.291830 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:03.291839 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:03.292305 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:03.293286 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:03.293312 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:03.294341 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:03.294345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:03.295411 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:03.295422 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:03.296071 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:03.296076 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:03.296361 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:03.297298 (+ 937us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:03.297329 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:03.299629 (+ 2300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:34:03.302359 (+ 2730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:03.302375 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:03.302393 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:03.302764 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:03.303240 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:03.303243 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:03.303244 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:03.303466 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:03.304040 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:34:03.304047 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:34:03.304440 (+ 393us) client_negotiation.cc:769] Sending connection context
0810 22:34:03.304624 (+ 184us) client_negotiation.cc:240] Negotiation successful
0810 22:34:03.305040 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":155,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.67@KRBTEST.COM" created.
I20260810 22:34:03.307850 22199 heartbeater.cc:347] Connected to a master server at 127.16.233.126:36183
I20260810 22:34:03.308188 22199 heartbeater.cc:464] Registering TS with master...
I20260810 22:34:03.308794 22199 heartbeater.cc:511] Master 127.16.233.126:36183 requested a full tablet report, sending...
I20260810 22:34:03.310436 21845 ts_manager.cc:194] Registered new tserver with Master: ce82c66759814208a320f1142be5ba16 (127.16.233.66:40959)
I20260810 22:34:03.311228 21845 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.66@KRBTEST.COM'} at 127.16.233.66:34085
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab.
Entry for principal kudu/127.16.233.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab.
I20260810 22:34:03.330263 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.67:0
--local_ip_for_outbound_sockets=127.16.233.67
--webserver_interface=127.16.233.67
--webserver_port=0
--tserver_master_addrs=127.16.233.126:36183
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab
--principal=kudu/127.16.233.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:46771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:34:03.458387 22205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:34:03.458760 22205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:34:03.458847 22205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:34:03.463228 22205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:34:03.463369 22205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:34:03.463479 22205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.67
I20260810 22:34:03.469895 22205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:46771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab
--principal=kudu/127.16.233.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.233.67
--webserver_port=0
--tserver_master_addrs=127.16.233.126:36183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.67
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:34:03.471166 22205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:34:03.472349 22205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:34:03.484215 22213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:03.484191 22211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:03.484215 22210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:03.484267 22205 server_base.cc:1034] running on GCE node
I20260810 22:34:03.485116 22205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:34:03.485975 22205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:34:03.487213 22205 hybrid_clock.cc:648] HybridClock initialized: now 1786401243487189 us; error 41 us; skew 500 ppm
Aug 10 22:34:03 dist-test-slave-77v9 krb5kdc[21796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401243, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:34:03.491046 22205 init.cc:377] Logged in from keytab as kudu/127.16.233.67@KRBTEST.COM (short username kudu)
I20260810 22:34:03.492069 22205 fs_manager.cc:359] Metadata directory not provided
I20260810 22:34:03.492192 22205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:34:03.492530 22205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:34:03.495257 22205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/minicluster-data/ts-2/data/instance:
uuid: "be9b168428a340e0869802e59d0ed66f"
format_stamp: "Formatted at 2026-08-10 22:34:03 on dist-test-slave-77v9"
server_key: "eedb432956e179d7a405ae68b0195501"
server_key_iv: "721d6158cddb9eaa9704c5afb58bfdd1"
server_key_version: "encryptionkey@0"
I20260810 22:34:03.495977 22205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/minicluster-data/ts-2/wal/instance:
uuid: "be9b168428a340e0869802e59d0ed66f"
format_stamp: "Formatted at 2026-08-10 22:34:03 on dist-test-slave-77v9"
server_key: "eedb432956e179d7a405ae68b0195501"
server_key_iv: "721d6158cddb9eaa9704c5afb58bfdd1"
server_key_version: "encryptionkey@0"
I20260810 22:34:03.500722 22205 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260810 22:34:03.503998 22220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:03.505728 22205 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260810 22:34:03.505925 22205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/minicluster-data/ts-2/wal
uuid: "be9b168428a340e0869802e59d0ed66f"
format_stamp: "Formatted at 2026-08-10 22:34:03 on dist-test-slave-77v9"
server_key: "eedb432956e179d7a405ae68b0195501"
server_key_iv: "721d6158cddb9eaa9704c5afb58bfdd1"
server_key_version: "encryptionkey@0"
I20260810 22:34:03.506086 22205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:34:03.524360 22205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:34:03.528290 22205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:34:03.528584 22205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:34:03.529444 22205 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:34:03.530471 22205 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:34:03.530537 22205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:03.530584 22205 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:34:03.530601 22205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:03.541867 22205 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.67:38629
I20260810 22:34:03.541945 22341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.67:38629 every 8 connection(s)
I20260810 22:34:03.542593 22205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
I20260810 22:34:03.550107 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 22205
I20260810 22:34:03.550266 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/minicluster-data/ts-2/wal/instance
I20260810 22:34:03.550596 17317 external_mini_cluster.cc:1428] Setting key c4f169037ccb53fd8e2f84429a337f2b
Aug 10 22:34:03 dist-test-slave-77v9 krb5kdc[21796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401243, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.67@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:34:03.558887 21886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:03.544644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:38061 (local address 127.16.233.126:36183)
0810 22:34:03.544880 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:03.544886 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:03.545839 (+ 953us) server_negotiation.cc:407] Connection header received
0810 22:34:03.547207 (+ 1368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:03.547211 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:03.547269 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:03.547358 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:03.548734 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:03.549377 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:03.551118 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:03.551532 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:03.551659 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:03.554854 (+ 3195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:03.554858 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:03.554861 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:03.554891 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:03.556460 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:03.557226 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:03.557229 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:03.557231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:03.557331 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:03.557677 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:03.557679 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:03.557680 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:03.557947 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:03.558033 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:03.558558 (+ 525us) server_negotiation.cc:299] Negotiation successful
0810 22:34:03.558691 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 22:34:03.559798 22344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:03.545132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:36183 (local address 127.16.233.67:38061)
0810 22:34:03.545662 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:03.545710 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:03.546984 (+ 1274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:03.547537 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:03.547544 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:03.548000 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:03.548531 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:03.548546 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:03.549518 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:03.549521 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:03.550960 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:03.550972 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:03.551711 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:03.551715 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:03.552027 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:03.552850 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:03.552875 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:03.554600 (+ 1725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:34:03.556595 (+ 1995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:03.556601 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:03.556617 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:03.557117 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:03.557428 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:03.557430 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:03.557431 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:03.557581 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:03.558046 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:34:03.558052 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:34:03.558317 (+ 265us) client_negotiation.cc:769] Sending connection context
0810 22:34:03.558490 (+ 173us) client_negotiation.cc:240] Negotiation successful
0810 22:34:03.558782 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":143,"threads_started":1}
I20260810 22:34:03.561476 22342 heartbeater.cc:347] Connected to a master server at 127.16.233.126:36183
I20260810 22:34:03.561770 22342 heartbeater.cc:464] Registering TS with master...
I20260810 22:34:03.562299 22342 heartbeater.cc:511] Master 127.16.233.126:36183 requested a full tablet report, sending...
I20260810 22:34:03.563597 21845 ts_manager.cc:194] Registered new tserver with Master: be9b168428a340e0869802e59d0ed66f (127.16.233.67:38629)
I20260810 22:34:03.564419 21845 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.67@KRBTEST.COM'} at 127.16.233.67:38061
I20260810 22:34:03.565584 17317 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 22:34:03.577862 21886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:03.568441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54460 (local address 127.16.233.126:36183)
0810 22:34:03.568592 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:03.568596 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:03.568987 (+ 391us) server_negotiation.cc:407] Connection header received
0810 22:34:03.569083 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:03.569086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:03.569135 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:03.569223 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:03.570131 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:03.570748 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:03.571672 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:03.571977 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:03.572102 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:03.573997 (+ 1895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:03.574001 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:03.574003 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:03.574033 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:03.575704 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:03.576182 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:03.576185 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:03.576187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:03.576290 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:03.576668 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:03.576671 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:03.576672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:03.577110 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:03.577241 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:03.577528 (+ 287us) server_negotiation.cc:299] Negotiation successful
0810 22:34:03.577650 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 22:34:03.581660 21845 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54460:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 22:34:03.585067 21845 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:34:03.603168 22357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:03.595988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38294 (local address 127.16.233.66:40959)
0810 22:34:03.596436 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:03.596451 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:03.596976 (+ 525us) server_negotiation.cc:407] Connection header received
0810 22:34:03.597208 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:03.597213 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:03.597390 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:03.597538 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:34:03.598451 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:03.599295 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:03.601502 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:03.602254 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:03.602415 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:03.602604 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:03.602857 (+ 253us) server_negotiation.cc:299] Negotiation successful
0810 22:34:03.602976 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":145,"threads_started":1}
I20260810 22:34:03.603173 22358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:03.595732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.66:40959 (local address 127.0.0.1:38294)
0810 22:34:03.596863 (+ 1131us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:03.596878 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:03.597057 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:03.597779 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:03.597782 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:03.597817 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:34:03.598326 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:03.598331 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:03.599462 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:03.599466 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:03.601352 (+ 1886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:03.601363 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:03.602419 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:03.602423 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:03.602567 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:03.602680 (+ 113us) client_negotiation.cc:769] Sending connection context
0810 22:34:03.602819 (+ 139us) client_negotiation.cc:240] Negotiation successful
0810 22:34:03.602954 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1024,"mutex_wait_us":1114,"spinlock_wait_cycles":8064,"thread_start_us":114,"threads_started":1}
I20260810 22:34:03.603790 22361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:03.597004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52800 (local address 127.16.233.65:42229)
0810 22:34:03.597563 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:03.597571 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:03.597596 (+ 25us) server_negotiation.cc:407] Connection header received
0810 22:34:03.597682 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:03.597690 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:03.597892 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:03.598038 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:34:03.598304 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:03.599543 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:03.601355 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:03.602307 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:03.603282 (+ 975us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:03.603477 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:03.603531 (+ 54us) server_negotiation.cc:299] Negotiation successful
0810 22:34:03.603625 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":112,"threads_started":1}
I20260810 22:34:03.604174 22355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:03.595814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58758 (local address 127.16.233.67:38629)
0810 22:34:03.596651 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:03.596664 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:03.597026 (+ 362us) server_negotiation.cc:407] Connection header received
0810 22:34:03.597122 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:03.597131 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:03.597391 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:03.597544 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:34:03.598273 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:03.599539 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:03.601294 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:03.602307 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:03.602446 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:03.602605 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:03.603444 (+ 839us) server_negotiation.cc:299] Negotiation successful
0810 22:34:03.604031 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":735,"thread_start_us":170,"threads_started":1}
I20260810 22:34:03.603204 22354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:03.595678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.67:38629 (local address 127.0.0.1:58758)
0810 22:34:03.596598 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:03.596668 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:03.596859 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:03.597768 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:03.597775 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:03.597816 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:34:03.598083 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:03.598091 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:03.599743 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:03.599746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:03.601137 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:03.601148 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:03.602671 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:03.602675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:03.602803 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:03.602826 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 22:34:03.602913 (+ 87us) client_negotiation.cc:240] Negotiation successful
0810 22:34:03.602980 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":714,"mutex_wait_us":37,"spinlock_wait_cycles":5888,"thread_start_us":145,"threads_started":1}
I20260810 22:34:03.603209 22359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:03.596777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.65:42229 (local address 127.0.0.1:52800)
0810 22:34:03.597169 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:03.597185 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:03.597297 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:03.598053 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:03.598057 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:03.598074 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:34:03.598172 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:03.598179 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:03.599685 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:03.599688 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:03.601214 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:03.601225 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:03.602478 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:03.602482 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:03.602582 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:03.602653 (+ 71us) client_negotiation.cc:769] Sending connection context
0810 22:34:03.602757 (+ 104us) client_negotiation.cc:240] Negotiation successful
0810 22:34:03.602947 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":11,"thread_start_us":153,"threads_started":1}
I20260810 22:34:03.606906 22127 tablet_service.cc:1467] Processing CreateTablet for tablet e7d151e5ef764e698d2c7aeffabc2982 (DEFAULT_TABLE table=test-table [id=a6974ab42012453cb18693578ca3fec3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:34:03.607118 22269 tablet_service.cc:1467] Processing CreateTablet for tablet e7d151e5ef764e698d2c7aeffabc2982 (DEFAULT_TABLE table=test-table [id=a6974ab42012453cb18693578ca3fec3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:34:03.607641 21983 tablet_service.cc:1467] Processing CreateTablet for tablet e7d151e5ef764e698d2c7aeffabc2982 (DEFAULT_TABLE table=test-table [id=a6974ab42012453cb18693578ca3fec3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:34:03.608247 22127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7d151e5ef764e698d2c7aeffabc2982. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:34:03.608271 22269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7d151e5ef764e698d2c7aeffabc2982. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:34:03.608716 21983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7d151e5ef764e698d2c7aeffabc2982. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:34:03.616182 22362 tablet_bootstrap.cc:492] T e7d151e5ef764e698d2c7aeffabc2982 P be9b168428a340e0869802e59d0ed66f: Bootstrap starting.
I20260810 22:34:03.617700 22364 tablet_bootstrap.cc:492] T e7d151e5ef764e698d2c7aeffabc2982 P ce82c66759814208a320f1142be5ba16: Bootstrap starting.
I20260810 22:34:03.618884 22362 tablet_bootstrap.cc:654] T e7d151e5ef764e698d2c7aeffabc2982 P be9b168428a340e0869802e59d0ed66f: Neither blocks nor log segments found. Creating new log.
I20260810 22:34:03.619786 22364 tablet_bootstrap.cc:654] T e7d151e5ef764e698d2c7aeffabc2982 P ce82c66759814208a320f1142be5ba16: Neither blocks nor log segments found. Creating new log.
I20260810 22:34:03.620489 22362 log.cc:824] T e7d151e5ef764e698d2c7aeffabc2982 P be9b168428a340e0869802e59d0ed66f: Log is configured to *not* fsync() on all Append() calls
I20260810 22:34:03.621315 22363 tablet_bootstrap.cc:492] T e7d151e5ef764e698d2c7aeffabc2982 P 8d97593349144b6ea5e7786351bf40fd: Bootstrap starting.
I20260810 22:34:03.621728 22364 log.cc:824] T e7d151e5ef764e698d2c7aeffabc2982 P ce82c66759814208a320f1142be5ba16: Log is configured to *not* fsync() on all Append() calls
I20260810 22:34:03.623358 22362 tablet_bootstrap.cc:492] T e7d151e5ef764e698d2c7aeffabc2982 P be9b168428a340e0869802e59d0ed66f: No bootstrap required, opened a new log
I20260810 22:34:03.623597 22362 ts_tablet_manager.cc:1397] T e7d151e5ef764e698d2c7aeffabc2982 P be9b168428a340e0869802e59d0ed66f: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260810 22:34:03.623721 22363 tablet_bootstrap.cc:654] T e7d151e5ef764e698d2c7aeffabc2982 P 8d97593349144b6ea5e7786351bf40fd: Neither blocks nor log segments found. Creating new log.
I20260810 22:34:03.624109 22364 tablet_bootstrap.cc:492] T e7d151e5ef764e698d2c7aeffabc2982 P ce82c66759814208a320f1142be5ba16: No bootstrap required, opened a new log
I20260810 22:34:03.624325 22364 ts_tablet_manager.cc:1397] T e7d151e5ef764e698d2c7aeffabc2982 P ce82c66759814208a320f1142be5ba16: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260810 22:34:03.624583 22363 log.cc:824] T e7d151e5ef764e698d2c7aeffabc2982 P 8d97593349144b6ea5e7786351bf40fd: Log is configured to *not* fsync() on all Append() calls
I20260810 22:34:03.626509 22363 tablet_bootstrap.cc:492] T e7d151e5ef764e698d2c7aeffabc2982 P 8d97593349144b6ea5e7786351bf40fd: No bootstrap required, opened a new log
I20260810 22:34:03.626747 22363 ts_tablet_manager.cc:1397] T e7d151e5ef764e698d2c7aeffabc2982 P 8d97593349144b6ea5e7786351bf40fd: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 22:34:03.628583 22364 raft_consensus.cc:348] T e7d151e5ef764e698d2c7aeffabc2982 P ce82c66759814208a320f1142be5ba16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be9b168428a340e0869802e59d0ed66f" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 38629 } } peers { permanent_uuid: "ce82c66759814208a320f1142be5ba16" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 40959 } } peers { permanent_uuid: "8d97593349144b6ea5e7786351bf40fd" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 42229 } }
I20260810 22:34:03.628583 22362 raft_consensus.cc:348] T e7d151e5ef764e698d2c7aeffabc2982 P be9b168428a340e0869802e59d0ed66f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be9b168428a340e0869802e59d0ed66f" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 38629 } } peers { permanent_uuid: "ce82c66759814208a320f1142be5ba16" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 40959 } } peers { permanent_uuid: "8d97593349144b6ea5e7786351bf40fd" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 42229 } }
I20260810 22:34:03.628979 22364 raft_consensus.cc:374] T e7d151e5ef764e698d2c7aeffabc2982 P ce82c66759814208a320f1142be5ba16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:34:03.628979 22362 raft_consensus.cc:374] T e7d151e5ef764e698d2c7aeffabc2982 P be9b168428a340e0869802e59d0ed66f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:34:03.629052 22364 raft_consensus.cc:729] T e7d151e5ef764e698d2c7aeffabc2982 P ce82c66759814208a320f1142be5ba16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce82c66759814208a320f1142be5ba16, State: Initialized, Role: FOLLOWER
I20260810 22:34:03.629052 22362 raft_consensus.cc:729] T e7d151e5ef764e698d2c7aeffabc2982 P be9b168428a340e0869802e59d0ed66f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be9b168428a340e0869802e59d0ed66f, State: Initialized, Role: FOLLOWER
I20260810 22:34:03.629604 22364 consensus_queue.cc:260] T e7d151e5ef764e698d2c7aeffabc2982 P ce82c66759814208a320f1142be5ba16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be9b168428a340e0869802e59d0ed66f" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 38629 } } peers { permanent_uuid: "ce82c66759814208a320f1142be5ba16" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 40959 } } peers { permanent_uuid: "8d97593349144b6ea5e7786351bf40fd" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 42229 } }
I20260810 22:34:03.629604 22362 consensus_queue.cc:260] T e7d151e5ef764e698d2c7aeffabc2982 P be9b168428a340e0869802e59d0ed66f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be9b168428a340e0869802e59d0ed66f" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 38629 } } peers { permanent_uuid: "ce82c66759814208a320f1142be5ba16" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 40959 } } peers { permanent_uuid: "8d97593349144b6ea5e7786351bf40fd" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 42229 } }
I20260810 22:34:03.630468 22199 heartbeater.cc:503] Master 127.16.233.126:36183 was elected leader, sending a full tablet report...
I20260810 22:34:03.630856 22364 ts_tablet_manager.cc:1428] T e7d151e5ef764e698d2c7aeffabc2982 P ce82c66759814208a320f1142be5ba16: Time spent starting tablet: real 0.006s user 0.004s sys 0.004s
I20260810 22:34:03.630857 22362 ts_tablet_manager.cc:1428] T e7d151e5ef764e698d2c7aeffabc2982 P be9b168428a340e0869802e59d0ed66f: Time spent starting tablet: real 0.007s user 0.004s sys 0.003s
I20260810 22:34:03.631095 22342 heartbeater.cc:503] Master 127.16.233.126:36183 was elected leader, sending a full tablet report...
I20260810 22:34:03.633138 22363 raft_consensus.cc:348] T e7d151e5ef764e698d2c7aeffabc2982 P 8d97593349144b6ea5e7786351bf40fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be9b168428a340e0869802e59d0ed66f" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 38629 } } peers { permanent_uuid: "ce82c66759814208a320f1142be5ba16" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 40959 } } peers { permanent_uuid: "8d97593349144b6ea5e7786351bf40fd" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 42229 } }
I20260810 22:34:03.633494 22363 raft_consensus.cc:374] T e7d151e5ef764e698d2c7aeffabc2982 P 8d97593349144b6ea5e7786351bf40fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:34:03.633569 22363 raft_consensus.cc:729] T e7d151e5ef764e698d2c7aeffabc2982 P 8d97593349144b6ea5e7786351bf40fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d97593349144b6ea5e7786351bf40fd, State: Initialized, Role: FOLLOWER
I20260810 22:34:03.634124 22363 consensus_queue.cc:260] T e7d151e5ef764e698d2c7aeffabc2982 P 8d97593349144b6ea5e7786351bf40fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be9b168428a340e0869802e59d0ed66f" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 38629 } } peers { permanent_uuid: "ce82c66759814208a320f1142be5ba16" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 40959 } } peers { permanent_uuid: "8d97593349144b6ea5e7786351bf40fd" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 42229 } }
I20260810 22:34:03.635011 22053 heartbeater.cc:503] Master 127.16.233.126:36183 was elected leader, sending a full tablet report...
I20260810 22:34:03.635452 22363 ts_tablet_manager.cc:1428] T e7d151e5ef764e698d2c7aeffabc2982 P 8d97593349144b6ea5e7786351bf40fd: Time spent starting tablet: real 0.009s user 0.004s sys 0.004s
I20260810 22:34:03.732350 22368 raft_consensus.cc:482] T e7d151e5ef764e698d2c7aeffabc2982 P be9b168428a340e0869802e59d0ed66f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:34:03.732654 22368 raft_consensus.cc:504] T e7d151e5ef764e698d2c7aeffabc2982 P be9b168428a340e0869802e59d0ed66f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be9b168428a340e0869802e59d0ed66f" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 38629 } } peers { permanent_uuid: "ce82c66759814208a320f1142be5ba16" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 40959 } } peers { permanent_uuid: "8d97593349144b6ea5e7786351bf40fd" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 42229 } }
I20260810 22:34:03.734202 22368 leader_election.cc:288] T e7d151e5ef764e698d2c7aeffabc2982 P be9b168428a340e0869802e59d0ed66f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce82c66759814208a320f1142be5ba16 (127.16.233.66:40959), 8d97593349144b6ea5e7786351bf40fd (127.16.233.65:42229)
I20260810 22:34:03.738848 22361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:03.734722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:46117 (local address 127.16.233.65:42229)
0810 22:34:03.734867 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:03.734872 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:03.734890 (+ 18us) server_negotiation.cc:407] Connection header received
0810 22:34:03.735080 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:03.735083 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:03.735139 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:03.735241 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:34:03.735563 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:03.736158 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:03.737666 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:03.738150 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:03.738310 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:03.738389 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:03.738609 (+ 220us) server_negotiation.cc:299] Negotiation successful
0810 22:34:03.738703 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 22:34:03.738852 22344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:03.734571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.65:42229 (local address 127.16.233.67:46117)
0810 22:34:03.734730 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:03.734750 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:03.734911 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:03.735258 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:03.735262 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:03.735281 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:34:03.735433 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:03.735440 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:03.736302 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:03.736306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:03.737506 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:03.737518 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:03.738305 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:03.738310 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:03.738438 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:03.738470 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 22:34:03.738586 (+ 116us) client_negotiation.cc:240] Negotiation successful
0810 22:34:03.738682 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":1}
I20260810 22:34:03.739213 22371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:03.734590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.66:40959 (local address 127.16.233.67:36411)
0810 22:34:03.735225 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:03.735239 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:03.735392 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:03.735720 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:03.735723 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:03.735741 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:34:03.735884 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:03.735891 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:03.736746 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:03.736750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:03.737905 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:03.737916 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:03.738698 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:03.738702 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:03.738827 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:03.738852 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 22:34:03.738965 (+ 113us) client_negotiation.cc:240] Negotiation successful
0810 22:34:03.739045 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"mutex_wait_us":163,"thread_start_us":205,"threads_started":1}
I20260810 22:34:03.739195 22357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:03.734735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:36411 (local address 127.16.233.66:40959)
0810 22:34:03.734879 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:03.734887 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:03.735356 (+ 469us) server_negotiation.cc:407] Connection header received
0810 22:34:03.735542 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:03.735546 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:03.735603 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:03.735703 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:34:03.736027 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:03.736560 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:03.738069 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:03.738547 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:03.738706 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:03.738763 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:03.738987 (+ 224us) server_negotiation.cc:299] Negotiation successful
0810 22:34:03.739072 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 22:34:03.739755 22004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7d151e5ef764e698d2c7aeffabc2982" candidate_uuid: "be9b168428a340e0869802e59d0ed66f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d97593349144b6ea5e7786351bf40fd" is_pre_election: true
I20260810 22:34:03.739929 22148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7d151e5ef764e698d2c7aeffabc2982" candidate_uuid: "be9b168428a340e0869802e59d0ed66f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce82c66759814208a320f1142be5ba16" is_pre_election: true
I20260810 22:34:03.740244 22004 raft_consensus.cc:2454] T e7d151e5ef764e698d2c7aeffabc2982 P 8d97593349144b6ea5e7786351bf40fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be9b168428a340e0869802e59d0ed66f in term 0.
I20260810 22:34:03.740243 22148 raft_consensus.cc:2454] T e7d151e5ef764e698d2c7aeffabc2982 P ce82c66759814208a320f1142be5ba16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be9b168428a340e0869802e59d0ed66f in term 0.
I20260810 22:34:03.741071 22224 leader_election.cc:302] T e7d151e5ef764e698d2c7aeffabc2982 P be9b168428a340e0869802e59d0ed66f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be9b168428a340e0869802e59d0ed66f, ce82c66759814208a320f1142be5ba16; no voters:
I20260810 22:34:03.741454 22368 raft_consensus.cc:2781] T e7d151e5ef764e698d2c7aeffabc2982 P be9b168428a340e0869802e59d0ed66f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:34:03.741559 22368 raft_consensus.cc:482] T e7d151e5ef764e698d2c7aeffabc2982 P be9b168428a340e0869802e59d0ed66f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:34:03.741596 22368 raft_consensus.cc:3037] T e7d151e5ef764e698d2c7aeffabc2982 P be9b168428a340e0869802e59d0ed66f [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:34:03.742934 22368 raft_consensus.cc:504] T e7d151e5ef764e698d2c7aeffabc2982 P be9b168428a340e0869802e59d0ed66f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be9b168428a340e0869802e59d0ed66f" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 38629 } } peers { permanent_uuid: "ce82c66759814208a320f1142be5ba16" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 40959 } } peers { permanent_uuid: "8d97593349144b6ea5e7786351bf40fd" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 42229 } }
I20260810 22:34:03.743458 22368 leader_election.cc:288] T e7d151e5ef764e698d2c7aeffabc2982 P be9b168428a340e0869802e59d0ed66f [CANDIDATE]: Term 1 election: Requested vote from peers ce82c66759814208a320f1142be5ba16 (127.16.233.66:40959), 8d97593349144b6ea5e7786351bf40fd (127.16.233.65:42229)
I20260810 22:34:03.743961 22148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7d151e5ef764e698d2c7aeffabc2982" candidate_uuid: "be9b168428a340e0869802e59d0ed66f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce82c66759814208a320f1142be5ba16"
I20260810 22:34:03.743954 22004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7d151e5ef764e698d2c7aeffabc2982" candidate_uuid: "be9b168428a340e0869802e59d0ed66f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d97593349144b6ea5e7786351bf40fd"
I20260810 22:34:03.744138 22148 raft_consensus.cc:3037] T e7d151e5ef764e698d2c7aeffabc2982 P ce82c66759814208a320f1142be5ba16 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:34:03.744138 22004 raft_consensus.cc:3037] T e7d151e5ef764e698d2c7aeffabc2982 P 8d97593349144b6ea5e7786351bf40fd [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:34:03.745492 22148 raft_consensus.cc:2454] T e7d151e5ef764e698d2c7aeffabc2982 P ce82c66759814208a320f1142be5ba16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be9b168428a340e0869802e59d0ed66f in term 1.
I20260810 22:34:03.745488 22004 raft_consensus.cc:2454] T e7d151e5ef764e698d2c7aeffabc2982 P 8d97593349144b6ea5e7786351bf40fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be9b168428a340e0869802e59d0ed66f in term 1.
I20260810 22:34:03.746009 22222 leader_election.cc:302] T e7d151e5ef764e698d2c7aeffabc2982 P be9b168428a340e0869802e59d0ed66f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 8d97593349144b6ea5e7786351bf40fd, be9b168428a340e0869802e59d0ed66f, ce82c66759814208a320f1142be5ba16; no voters:
I20260810 22:34:03.746236 22368 raft_consensus.cc:2781] T e7d151e5ef764e698d2c7aeffabc2982 P be9b168428a340e0869802e59d0ed66f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:34:03.746546 22368 raft_consensus.cc:686] T e7d151e5ef764e698d2c7aeffabc2982 P be9b168428a340e0869802e59d0ed66f [term 1 LEADER]: Becoming Leader. State: Replica: be9b168428a340e0869802e59d0ed66f, State: Running, Role: LEADER
I20260810 22:34:03.746917 22368 consensus_queue.cc:237] T e7d151e5ef764e698d2c7aeffabc2982 P be9b168428a340e0869802e59d0ed66f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be9b168428a340e0869802e59d0ed66f" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 38629 } } peers { permanent_uuid: "ce82c66759814208a320f1142be5ba16" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 40959 } } peers { permanent_uuid: "8d97593349144b6ea5e7786351bf40fd" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 42229 } }
I20260810 22:34:03.750874 21846 catalog_manager.cc:5495] T e7d151e5ef764e698d2c7aeffabc2982 P be9b168428a340e0869802e59d0ed66f reported cstate change: term changed from 0 to 1, leader changed from <none> to be9b168428a340e0869802e59d0ed66f (127.16.233.67). New cstate: current_term: 1 leader_uuid: "be9b168428a340e0869802e59d0ed66f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be9b168428a340e0869802e59d0ed66f" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 38629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce82c66759814208a320f1142be5ba16" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 40959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d97593349144b6ea5e7786351bf40fd" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 42229 } health_report { overall_health: UNKNOWN } } }
W20260810 22:34:03.779917 22054 tablet.cc:2369] T e7d151e5ef764e698d2c7aeffabc2982 P 8d97593349144b6ea5e7786351bf40fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 22:34:03.787405 22200 tablet.cc:2369] T e7d151e5ef764e698d2c7aeffabc2982 P ce82c66759814208a320f1142be5ba16: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 22:34:03.793678 22343 tablet.cc:2369] T e7d151e5ef764e698d2c7aeffabc2982 P be9b168428a340e0869802e59d0ed66f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:34:03.799763 22355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:03.795843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58774 (local address 127.16.233.67:38629)
0810 22:34:03.796041 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:03.796046 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:03.796067 (+ 21us) server_negotiation.cc:407] Connection header received
0810 22:34:03.796142 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:03.796144 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:03.796198 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:03.796321 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:34:03.796604 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:03.797152 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:03.798130 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:03.798359 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:03.798470 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:03.798830 (+ 360us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:34:03.799294 (+ 464us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:34:03.799438 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:03.799508 (+ 70us) server_negotiation.cc:299] Negotiation successful
0810 22:34:03.799572 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260810 22:34:03.806515 22148 raft_consensus.cc:1261] T e7d151e5ef764e698d2c7aeffabc2982 P ce82c66759814208a320f1142be5ba16 [term 1 FOLLOWER]: Refusing update from remote peer be9b168428a340e0869802e59d0ed66f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:34:03.806543 22004 raft_consensus.cc:1261] T e7d151e5ef764e698d2c7aeffabc2982 P 8d97593349144b6ea5e7786351bf40fd [term 1 FOLLOWER]: Refusing update from remote peer be9b168428a340e0869802e59d0ed66f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:34:03.807461 22372 consensus_queue.cc:1035] T e7d151e5ef764e698d2c7aeffabc2982 P be9b168428a340e0869802e59d0ed66f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d97593349144b6ea5e7786351bf40fd" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 42229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:34:03.807703 22368 consensus_queue.cc:1035] T e7d151e5ef764e698d2c7aeffabc2982 P be9b168428a340e0869802e59d0ed66f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce82c66759814208a320f1142be5ba16" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 40959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:34:03.815470 22375 mvcc.cc:205] Tried to move back new op lower bound from 7317099494622814208 to 7317099494390333440. Current Snapshot: MvccSnapshot[applied={T|T < 7317099494622814208}]
I20260810 22:34:03.820528 22376 mvcc.cc:205] Tried to move back new op lower bound from 7317099494622814208 to 7317099494390333440. Current Snapshot: MvccSnapshot[applied={T|T < 7317099494622814208 or (T in {7317099494622814208})}]
I20260810 22:34:03.820863 22377 mvcc.cc:205] Tried to move back new op lower bound from 7317099494622814208 to 7317099494390333440. Current Snapshot: MvccSnapshot[applied={T|T < 7317099494622814208}]
I20260810 22:34:03.826903 21844 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:34:03.827184 21844 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:34:03.830786 21844 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f2c09e72f1d44f109a30ee9f151676ce: Sending DeleteTablet for 3 replicas of tablet e7d151e5ef764e698d2c7aeffabc2982
I20260810 22:34:03.831982 21983 tablet_service.cc:1514] Processing DeleteTablet for tablet e7d151e5ef764e698d2c7aeffabc2982 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:34:03 UTC) from {username='kudu', principal='kudu/127.16.233.126@KRBTEST.COM'} at 127.0.0.1:52800
I20260810 22:34:03.831985 22127 tablet_service.cc:1514] Processing DeleteTablet for tablet e7d151e5ef764e698d2c7aeffabc2982 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:34:03 UTC) from {username='kudu', principal='kudu/127.16.233.126@KRBTEST.COM'} at 127.0.0.1:38294
I20260810 22:34:03.832060 22269 tablet_service.cc:1514] Processing DeleteTablet for tablet e7d151e5ef764e698d2c7aeffabc2982 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:34:03 UTC) from {username='kudu', principal='kudu/127.16.233.126@KRBTEST.COM'} at 127.0.0.1:58758
I20260810 22:34:03.832957 22386 tablet_replica.cc:330] T e7d151e5ef764e698d2c7aeffabc2982 P 8d97593349144b6ea5e7786351bf40fd: stopping tablet replica
I20260810 22:34:03.833323 22387 tablet_replica.cc:330] T e7d151e5ef764e698d2c7aeffabc2982 P be9b168428a340e0869802e59d0ed66f: stopping tablet replica
I20260810 22:34:03.833441 22386 raft_consensus.cc:2229] T e7d151e5ef764e698d2c7aeffabc2982 P 8d97593349144b6ea5e7786351bf40fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:34:03.833591 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 21913
I20260810 22:34:03.834007 22387 raft_consensus.cc:2229] T e7d151e5ef764e698d2c7aeffabc2982 P be9b168428a340e0869802e59d0ed66f [term 1 LEADER]: Raft consensus shutting down.
I20260810 22:34:03.834587 22387 raft_consensus.cc:2258] T e7d151e5ef764e698d2c7aeffabc2982 P be9b168428a340e0869802e59d0ed66f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:34:03.838114 22384 tablet_replica.cc:330] T e7d151e5ef764e698d2c7aeffabc2982 P ce82c66759814208a320f1142be5ba16: stopping tablet replica
I20260810 22:34:03.838620 22384 raft_consensus.cc:2229] T e7d151e5ef764e698d2c7aeffabc2982 P ce82c66759814208a320f1142be5ba16 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:34:03.839066 22384 raft_consensus.cc:2258] T e7d151e5ef764e698d2c7aeffabc2982 P ce82c66759814208a320f1142be5ba16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:34:03.841459 22384 ts_tablet_manager.cc:1905] T e7d151e5ef764e698d2c7aeffabc2982 P ce82c66759814208a320f1142be5ba16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:34:03.841459 22387 ts_tablet_manager.cc:1905] T e7d151e5ef764e698d2c7aeffabc2982 P be9b168428a340e0869802e59d0ed66f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 22:34:03.845239 21827 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.233.65:42229 (error 108)
I20260810 22:34:03.845431 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 22058
I20260810 22:34:03.855228 22354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:03.854735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.65:42229 (local address 127.0.0.1:52810)
0810 22:34:03.855007 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:03.855108 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.65:42229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175}
W20260810 22:34:03.855513 21827 catalog_manager.cc:4584] TS 8d97593349144b6ea5e7786351bf40fd (127.16.233.65:42229): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e7d151e5ef764e698d2c7aeffabc2982: Network error: Client connection negotiation failed: client connection to 127.16.233.65:42229: connect: Connection refused (error 111)
I20260810 22:34:03.857818 22387 ts_tablet_manager.cc:1918] T e7d151e5ef764e698d2c7aeffabc2982 P be9b168428a340e0869802e59d0ed66f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:34:03.857970 22387 log.cc:1196] T e7d151e5ef764e698d2c7aeffabc2982 P be9b168428a340e0869802e59d0ed66f: Deleting WAL directory at /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786401215402979-17317-0/minicluster-data/ts-2/wal/wals/e7d151e5ef764e698d2c7aeffabc2982
I20260810 22:34:03.858362 22387 ts_tablet_manager.cc:1939] T e7d151e5ef764e698d2c7aeffabc2982 P be9b168428a340e0869802e59d0ed66f: Deleting consensus metadata
I20260810 22:34:03.859956 21827 catalog_manager.cc:4841] TS be9b168428a340e0869802e59d0ed66f (127.16.233.67:38629): tablet e7d151e5ef764e698d2c7aeffabc2982 (table test-table [id=a6974ab42012453cb18693578ca3fec3]) successfully deleted
I20260810 22:34:03.864200 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 22205
I20260810 22:34:03.864864 22354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:03.864451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.66:40959 (local address 127.0.0.1:38304)
0810 22:34:03.864663 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:03.864725 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.66:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260810 22:34:03.873137 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 21810
2026-08-10T22:34:03Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2712 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-10T22:34:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:34:03Z Disabled control of system clock
I20260810 22:34:03.912690 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786401215402979-17317-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786401215402979-17317-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786401215402979-17317-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:42743
--webserver_interface=127.16.233.126
--webserver_port=0
--builtin_ntp_servers=127.16.233.84:37451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.126:42743
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 22:34:04.037107 22392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:34:04.037448 22392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:34:04.037490 22392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:34:04.041987 22392 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:34:04.042073 22392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:34:04.042088 22392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:34:04.042107 22392 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:34:04.042124 22392 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:34:04.047454 22392 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:37451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786401215402979-17317-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786401215402979-17317-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.126:42743
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:42743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786401215402979-17317-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:34:04.048848 22392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:34:04.050601 22392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:34:04.061017 22397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:04.061057 22400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:04.061017 22398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:04.061488 22392 server_base.cc:1034] running on GCE node
I20260810 22:34:04.062264 22392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:34:04.063709 22392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:34:04.064963 22392 hybrid_clock.cc:648] HybridClock initialized: now 1786401244064928 us; error 58 us; skew 500 ppm
I20260810 22:34:04.067507 22392 webserver.cc:460] Webserver started at http://127.16.233.126:42429/ using document root <none> and password file <none>
I20260810 22:34:04.068213 22392 fs_manager.cc:359] Metadata directory not provided
I20260810 22:34:04.068336 22392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:34:04.068612 22392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:34:04.071496 22392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786401215402979-17317-0/minicluster-data/master-0/data/instance:
uuid: "2e69d3ea627342978b7e809974aa55da"
format_stamp: "Formatted at 2026-08-10 22:34:04 on dist-test-slave-77v9"
server_key: "44e491a5800a9b0279b8686f4b4d6a42"
server_key_iv: "33bf49b7b57c58aac0a0b1d529f5fc6a"
server_key_version: "encryptionkey@0"
I20260810 22:34:04.072296 22392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786401215402979-17317-0/minicluster-data/master-0/wal/instance:
uuid: "2e69d3ea627342978b7e809974aa55da"
format_stamp: "Formatted at 2026-08-10 22:34:04 on dist-test-slave-77v9"
server_key: "44e491a5800a9b0279b8686f4b4d6a42"
server_key_iv: "33bf49b7b57c58aac0a0b1d529f5fc6a"
server_key_version: "encryptionkey@0"
I20260810 22:34:04.076949 22392 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.001s
I20260810 22:34:04.079808 22406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:04.081326 22392 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260810 22:34:04.081498 22392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786401215402979-17317-0/minicluster-data/master-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786401215402979-17317-0/minicluster-data/master-0/wal
uuid: "2e69d3ea627342978b7e809974aa55da"
format_stamp: "Formatted at 2026-08-10 22:34:04 on dist-test-slave-77v9"
server_key: "44e491a5800a9b0279b8686f4b4d6a42"
server_key_iv: "33bf49b7b57c58aac0a0b1d529f5fc6a"
server_key_version: "encryptionkey@0"
I20260810 22:34:04.081640 22392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786401215402979-17317-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786401215402979-17317-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786401215402979-17317-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:34:04.086864 22392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:34:04.087579 22392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:34:04.087797 22392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:34:04.096196 22392 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.126:42743
I20260810 22:34:04.096201 22465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.126:42743 every 8 connection(s)
I20260810 22:34:04.097380 22392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
I20260810 22:34:04.101145 22466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:34:04.101191 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 22392
I20260810 22:34:04.101337 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786401215402979-17317-0/minicluster-data/master-0/wal/instance
I20260810 22:34:04.101656 17317 external_mini_cluster.cc:1428] Setting key 6ecebb8faa20b1285392424561674068
I20260810 22:34:04.108764 22469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:04.103320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36280 (local address 127.16.233.126:42743)
0810 22:34:04.104085 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:04.104107 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:04.104160 (+ 53us) server_negotiation.cc:407] Connection header received
0810 22:34:04.105081 (+ 921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:04.105109 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:04.105631 (+ 522us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:04.106067 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:04.107014 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:04.107021 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:04.107036 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:34:04.107049 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:04.107120 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:34:04.107145 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:04.107318 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:04.107461 (+ 143us) server_negotiation.cc:299] Negotiation successful
0810 22:34:04.107736 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":144,"threads_started":1}
I20260810 22:34:04.109560 22466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e69d3ea627342978b7e809974aa55da: Bootstrap starting.
I20260810 22:34:04.111979 22466 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e69d3ea627342978b7e809974aa55da: Neither blocks nor log segments found. Creating new log.
I20260810 22:34:04.112866 22466 log.cc:824] T 00000000000000000000000000000000 P 2e69d3ea627342978b7e809974aa55da: Log is configured to *not* fsync() on all Append() calls
I20260810 22:34:04.115166 22466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e69d3ea627342978b7e809974aa55da: No bootstrap required, opened a new log
I20260810 22:34:04.119088 22466 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e69d3ea627342978b7e809974aa55da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e69d3ea627342978b7e809974aa55da" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 42743 } }
I20260810 22:34:04.119395 22466 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e69d3ea627342978b7e809974aa55da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:34:04.119436 22466 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e69d3ea627342978b7e809974aa55da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e69d3ea627342978b7e809974aa55da, State: Initialized, Role: FOLLOWER
I20260810 22:34:04.119954 22466 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e69d3ea627342978b7e809974aa55da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e69d3ea627342978b7e809974aa55da" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 42743 } }
I20260810 22:34:04.120093 22466 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e69d3ea627342978b7e809974aa55da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:34:04.120194 22466 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e69d3ea627342978b7e809974aa55da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:34:04.120345 22466 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e69d3ea627342978b7e809974aa55da [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:34:04.121583 22466 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e69d3ea627342978b7e809974aa55da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e69d3ea627342978b7e809974aa55da" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 42743 } }
I20260810 22:34:04.121963 22466 leader_election.cc:302] T 00000000000000000000000000000000 P 2e69d3ea627342978b7e809974aa55da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e69d3ea627342978b7e809974aa55da; no voters:
I20260810 22:34:04.122335 22466 leader_election.cc:288] T 00000000000000000000000000000000 P 2e69d3ea627342978b7e809974aa55da [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:34:04.122505 22471 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e69d3ea627342978b7e809974aa55da [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:34:04.122776 22471 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e69d3ea627342978b7e809974aa55da [term 1 LEADER]: Becoming Leader. State: Replica: 2e69d3ea627342978b7e809974aa55da, State: Running, Role: LEADER
I20260810 22:34:04.123116 22471 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e69d3ea627342978b7e809974aa55da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e69d3ea627342978b7e809974aa55da" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 42743 } }
I20260810 22:34:04.123739 22466 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e69d3ea627342978b7e809974aa55da [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:34:04.124981 22472 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e69d3ea627342978b7e809974aa55da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e69d3ea627342978b7e809974aa55da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e69d3ea627342978b7e809974aa55da" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 42743 } } }
I20260810 22:34:04.125123 22472 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e69d3ea627342978b7e809974aa55da [sys.catalog]: This master's current role is: LEADER
I20260810 22:34:04.125412 22473 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e69d3ea627342978b7e809974aa55da [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e69d3ea627342978b7e809974aa55da. Latest consensus state: current_term: 1 leader_uuid: "2e69d3ea627342978b7e809974aa55da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e69d3ea627342978b7e809974aa55da" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 42743 } } }
I20260810 22:34:04.125525 22473 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e69d3ea627342978b7e809974aa55da [sys.catalog]: This master's current role is: LEADER
I20260810 22:34:04.125618 22480 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:34:04.129102 22480 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:34:04.135303 22480 catalog_manager.cc:1347] Generated new cluster ID: 60885aae13e64e46bca36f1a6b831d36
I20260810 22:34:04.135406 22480 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:34:04.146840 22480 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:34:04.147364 22480 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:34:04.167186 22480 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e69d3ea627342978b7e809974aa55da: Generated new TSK 0
I20260810 22:34:04.168206 22480 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:34:04.176393 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786401215402979-17317-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786401215402979-17317-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786401215402979-17317-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.65:0
--local_ip_for_outbound_sockets=127.16.233.65
--webserver_interface=127.16.233.65
--webserver_port=0
--tserver_master_addrs=127.16.233.126:42743
--builtin_ntp_servers=127.16.233.84:37451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 22:34:04.300676 22491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:34:04.301054 22491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:34:04.301141 22491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:34:04.305702 22491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:34:04.305786 22491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:34:04.305851 22491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.65
I20260810 22:34:04.311045 22491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:37451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786401215402979-17317-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786401215402979-17317-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.233.65
--webserver_port=0
--tserver_master_addrs=127.16.233.126:42743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22491
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.65
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786401215402979-17317-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:34:04.312315 22491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:34:04.313989 22491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:34:04.324882 22500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:04.325029 22491 server_base.cc:1034] running on GCE node
W20260810 22:34:04.324882 22498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:04.325135 22497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:04.325713 22491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:34:04.326520 22491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:34:04.327767 22491 hybrid_clock.cc:648] HybridClock initialized: now 1786401244327717 us; error 73 us; skew 500 ppm
I20260810 22:34:04.330236 22491 webserver.cc:460] Webserver started at http://127.16.233.65:38815/ using document root <none> and password file <none>
I20260810 22:34:04.330942 22491 fs_manager.cc:359] Metadata directory not provided
I20260810 22:34:04.331063 22491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:34:04.331334 22491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:34:04.333809 22491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786401215402979-17317-0/minicluster-data/ts-0/data/instance:
uuid: "1d284c2b05824f99a2ec002051e7fc30"
format_stamp: "Formatted at 2026-08-10 22:34:04 on dist-test-slave-77v9"
server_key: "9d814c52036b2c6813fbcf798ab4321a"
server_key_iv: "fd20a74790e161923e3ce5154fca91b8"
server_key_version: "encryptionkey@0"
I20260810 22:34:04.334576 22491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786401215402979-17317-0/minicluster-data/ts-0/wal/instance:
uuid: "1d284c2b05824f99a2ec002051e7fc30"
format_stamp: "Formatted at 2026-08-10 22:34:04 on dist-test-slave-77v9"
server_key: "9d814c52036b2c6813fbcf798ab4321a"
server_key_iv: "fd20a74790e161923e3ce5154fca91b8"
server_key_version: "encryptionkey@0"
I20260810 22:34:04.339366 22491 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 22:34:04.342767 22506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:04.344342 22491 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 22:34:04.344537 22491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786401215402979-17317-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786401215402979-17317-0/minicluster-data/ts-0/wal
uuid: "1d284c2b05824f99a2ec002051e7fc30"
format_stamp: "Formatted at 2026-08-10 22:34:04 on dist-test-slave-77v9"
server_key: "9d814c52036b2c6813fbcf798ab4321a"
server_key_iv: "fd20a74790e161923e3ce5154fca91b8"
server_key_version: "encryptionkey@0"
I20260810 22:34:04.344691 22491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786401215402979-17317-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786401215402979-17317-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786401215402979-17317-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:34:04.350423 22491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:34:04.351269 22491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:34:04.351528 22491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:34:04.352255 22491 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:34:04.353504 22491 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:34:04.353591 22491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:04.353685 22491 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:34:04.353736 22491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:04.364279 22491 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.65:38099
I20260810 22:34:04.364323 22630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.65:38099 every 8 connection(s)
I20260810 22:34:04.365391 22491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
I20260810 22:34:04.372938 22469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:04.367623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:41777 (local address 127.16.233.126:42743)
0810 22:34:04.367799 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:04.367803 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:04.369083 (+ 1280us) server_negotiation.cc:407] Connection header received
0810 22:34:04.370728 (+ 1645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:04.370732 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:04.370782 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:04.370904 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:04.372304 (+ 1400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:04.372308 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:04.372311 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:34:04.372327 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:04.372362 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:34:04.372374 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:04.372491 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:04.372666 (+ 175us) server_negotiation.cc:299] Negotiation successful
0810 22:34:04.372736 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 22:34:04.373911 22633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:04.368098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:42743 (local address 127.16.233.65:41777)
0810 22:34:04.368865 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:04.368956 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:04.370489 (+ 1533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:04.371080 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:04.371087 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:04.371198 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:04.371975 (+ 777us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 22:34:04.371977 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:04.372021 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 22:34:04.372109 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:34:04.372482 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:34:04.372492 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:34:04.372500 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 22:34:04.372703 (+ 203us) client_negotiation.cc:240] Negotiation successful
0810 22:34:04.373008 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":149,"threads_started":1}
I20260810 22:34:04.375484 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 22491
I20260810 22:34:04.375471 22631 heartbeater.cc:347] Connected to a master server at 127.16.233.126:42743
I20260810 22:34:04.375648 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786401215402979-17317-0/minicluster-data/ts-0/wal/instance
I20260810 22:34:04.375753 22631 heartbeater.cc:464] Registering TS with master...
I20260810 22:34:04.375980 17317 external_mini_cluster.cc:1428] Setting key b7ab66782941064239d1e553a09e1830
I20260810 22:34:04.376278 22631 heartbeater.cc:511] Master 127.16.233.126:42743 requested a full tablet report, sending...
I20260810 22:34:04.378194 22424 ts_manager.cc:194] Registered new tserver with Master: 1d284c2b05824f99a2ec002051e7fc30 (127.16.233.65:38099)
I20260810 22:34:04.378288 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786401215402979-17317-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786401215402979-17317-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786401215402979-17317-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.66:0
--local_ip_for_outbound_sockets=127.16.233.66
--webserver_interface=127.16.233.66
--webserver_port=0
--tserver_master_addrs=127.16.233.126:42743
--builtin_ntp_servers=127.16.233.84:37451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 22:34:04.500495 22634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:34:04.500892 22634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:34:04.501005 22634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:34:04.505417 22634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:34:04.505546 22634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:34:04.505656 22634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.66
I20260810 22:34:04.510788 22634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:37451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786401215402979-17317-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786401215402979-17317-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.233.66
--webserver_port=0
--tserver_master_addrs=127.16.233.126:42743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.66
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786401215402979-17317-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:34:04.512058 22634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:34:04.513623 22634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 22:34:04.524665 22634 server_base.cc:1034] running on GCE node
W20260810 22:34:04.524595 22642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:04.524623 22640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:04.524606 22639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:04.525389 22634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:34:04.526266 22634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:34:04.527486 22634 hybrid_clock.cc:648] HybridClock initialized: now 1786401244527448 us; error 53 us; skew 500 ppm
I20260810 22:34:04.529814 22634 webserver.cc:460] Webserver started at http://127.16.233.66:34619/ using document root <none> and password file <none>
I20260810 22:34:04.530486 22634 fs_manager.cc:359] Metadata directory not provided
I20260810 22:34:04.530591 22634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:34:04.530850 22634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:34:04.533314 22634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786401215402979-17317-0/minicluster-data/ts-1/data/instance:
uuid: "6017e08b7db64e29bd259c112d69e2dd"
format_stamp: "Formatted at 2026-08-10 22:34:04 on dist-test-slave-77v9"
server_key: "c3fe061e3fcfa0f4458b1d6e3dbebed6"
server_key_iv: "f500a529b853d9f9d7b7e51ca262e466"
server_key_version: "encryptionkey@0"
I20260810 22:34:04.533960 22634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786401215402979-17317-0/minicluster-data/ts-1/wal/instance:
uuid: "6017e08b7db64e29bd259c112d69e2dd"
format_stamp: "Formatted at 2026-08-10 22:34:04 on dist-test-slave-77v9"
server_key: "c3fe061e3fcfa0f4458b1d6e3dbebed6"
server_key_iv: "f500a529b853d9f9d7b7e51ca262e466"
server_key_version: "encryptionkey@0"
I20260810 22:34:04.538539 22634 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 22:34:04.541726 22648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:04.543263 22634 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 22:34:04.543481 22634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786401215402979-17317-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786401215402979-17317-0/minicluster-data/ts-1/wal
uuid: "6017e08b7db64e29bd259c112d69e2dd"
format_stamp: "Formatted at 2026-08-10 22:34:04 on dist-test-slave-77v9"
server_key: "c3fe061e3fcfa0f4458b1d6e3dbebed6"
server_key_iv: "f500a529b853d9f9d7b7e51ca262e466"
server_key_version: "encryptionkey@0"
I20260810 22:34:04.543637 22634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786401215402979-17317-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786401215402979-17317-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786401215402979-17317-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:34:04.549108 22634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:34:04.549954 22634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:34:04.550169 22634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:34:04.550776 22634 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:34:04.551788 22634 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:34:04.551873 22634 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:04.551946 22634 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:34:04.551992 22634 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:04.563314 22634 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.66:37151
I20260810 22:34:04.563393 22771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.66:37151 every 8 connection(s)
I20260810 22:34:04.564514 22634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
I20260810 22:34:04.567162 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 22634
I20260810 22:34:04.567308 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786401215402979-17317-0/minicluster-data/ts-1/wal/instance
I20260810 22:34:04.567598 17317 external_mini_cluster.cc:1428] Setting key e9d42c3415e58ade6fa13744179494fc
I20260810 22:34:04.570237 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786401215402979-17317-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786401215402979-17317-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786401215402979-17317-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.67:0
--local_ip_for_outbound_sockets=127.16.233.67
--webserver_interface=127.16.233.67
--webserver_port=0
--tserver_master_addrs=127.16.233.126:42743
--builtin_ntp_servers=127.16.233.84:37451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260810 22:34:04.572279 22469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:04.566864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:46263 (local address 127.16.233.126:42743)
0810 22:34:04.567066 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:04.567071 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:04.568265 (+ 1194us) server_negotiation.cc:407] Connection header received
0810 22:34:04.570006 (+ 1741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:04.570012 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:04.570059 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:04.570188 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:04.571585 (+ 1397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:04.571593 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:04.571599 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:34:04.571617 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:04.571648 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:34:04.571661 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:04.571766 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:04.572013 (+ 247us) server_negotiation.cc:299] Negotiation successful
0810 22:34:04.572075 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 22:34:04.573568 22775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:04.567249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:42743 (local address 127.16.233.66:46263)
0810 22:34:04.568092 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:04.568149 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:04.569737 (+ 1588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:04.570387 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:04.570398 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:04.570528 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:04.571290 (+ 762us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 22:34:04.571291 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:04.571337 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 22:34:04.571412 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:34:04.571819 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:34:04.571833 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:34:04.571843 (+ 10us) client_negotiation.cc:769] Sending connection context
0810 22:34:04.572046 (+ 203us) client_negotiation.cc:240] Negotiation successful
0810 22:34:04.572381 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":156,"threads_started":1}
I20260810 22:34:04.575062 22772 heartbeater.cc:347] Connected to a master server at 127.16.233.126:42743
I20260810 22:34:04.575333 22772 heartbeater.cc:464] Registering TS with master...
I20260810 22:34:04.575975 22772 heartbeater.cc:511] Master 127.16.233.126:42743 requested a full tablet report, sending...
I20260810 22:34:04.577353 22424 ts_manager.cc:194] Registered new tserver with Master: 6017e08b7db64e29bd259c112d69e2dd (127.16.233.66:37151)
W20260810 22:34:04.697160 22776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:34:04.697490 22776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:34:04.697532 22776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:34:04.701867 22776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:34:04.701947 22776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:34:04.702009 22776 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.67
I20260810 22:34:04.707145 22776 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:37451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786401215402979-17317-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786401215402979-17317-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.233.67
--webserver_port=0
--tserver_master_addrs=127.16.233.126:42743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.67
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786401215402979-17317-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:34:04.708504 22776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:34:04.710125 22776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:34:04.721247 22782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:04.721349 22776 server_base.cc:1034] running on GCE node
W20260810 22:34:04.721308 22781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:04.721282 22784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:04.721987 22776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:34:04.722738 22776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:34:04.723977 22776 hybrid_clock.cc:648] HybridClock initialized: now 1786401244723934 us; error 58 us; skew 500 ppm
I20260810 22:34:04.726439 22776 webserver.cc:460] Webserver started at http://127.16.233.67:34489/ using document root <none> and password file <none>
I20260810 22:34:04.727125 22776 fs_manager.cc:359] Metadata directory not provided
I20260810 22:34:04.727250 22776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:34:04.727514 22776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:34:04.729954 22776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786401215402979-17317-0/minicluster-data/ts-2/data/instance:
uuid: "23ff9fdd63ae4a48b56ddabfae058e03"
format_stamp: "Formatted at 2026-08-10 22:34:04 on dist-test-slave-77v9"
server_key: "ab0ace89b114895302f1f2936af50ba0"
server_key_iv: "2e875a306451707b0efb7b6f0350f2e1"
server_key_version: "encryptionkey@0"
I20260810 22:34:04.730661 22776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786401215402979-17317-0/minicluster-data/ts-2/wal/instance:
uuid: "23ff9fdd63ae4a48b56ddabfae058e03"
format_stamp: "Formatted at 2026-08-10 22:34:04 on dist-test-slave-77v9"
server_key: "ab0ace89b114895302f1f2936af50ba0"
server_key_iv: "2e875a306451707b0efb7b6f0350f2e1"
server_key_version: "encryptionkey@0"
I20260810 22:34:04.734907 22776 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 22:34:04.737923 22790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:04.739454 22776 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:34:04.739624 22776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786401215402979-17317-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786401215402979-17317-0/minicluster-data/ts-2/wal
uuid: "23ff9fdd63ae4a48b56ddabfae058e03"
format_stamp: "Formatted at 2026-08-10 22:34:04 on dist-test-slave-77v9"
server_key: "ab0ace89b114895302f1f2936af50ba0"
server_key_iv: "2e875a306451707b0efb7b6f0350f2e1"
server_key_version: "encryptionkey@0"
I20260810 22:34:04.739779 22776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786401215402979-17317-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786401215402979-17317-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786401215402979-17317-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:34:04.745146 22776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:34:04.745916 22776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:34:04.746134 22776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:34:04.746809 22776 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:34:04.747871 22776 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:34:04.747951 22776 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:04.748023 22776 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:34:04.748078 22776 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:04.759632 22776 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.67:33533
I20260810 22:34:04.759716 22912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.67:33533 every 8 connection(s)
I20260810 22:34:04.760989 22776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
I20260810 22:34:04.768067 22469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:04.763275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:54213 (local address 127.16.233.126:42743)
0810 22:34:04.763458 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:04.763463 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:04.764670 (+ 1207us) server_negotiation.cc:407] Connection header received
0810 22:34:04.766152 (+ 1482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:04.766155 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:04.766206 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:04.766323 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:04.767559 (+ 1236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:04.767562 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:04.767564 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:34:04.767578 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:04.767599 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:34:04.767607 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:04.767712 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:04.767865 (+ 153us) server_negotiation.cc:299] Negotiation successful
0810 22:34:04.767918 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260810 22:34:04.769124 22915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:04.763683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:42743 (local address 127.16.233.67:54213)
0810 22:34:04.764451 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:04.764496 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:04.765903 (+ 1407us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:04.766515 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:04.766522 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:04.766621 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:04.767226 (+ 605us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 22:34:04.767228 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:04.767273 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 22:34:04.767366 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:34:04.767690 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:34:04.767699 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:34:04.767705 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 22:34:04.767914 (+ 209us) client_negotiation.cc:240] Negotiation successful
0810 22:34:04.768174 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":127,"threads_started":1}
I20260810 22:34:04.769469 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 22776
I20260810 22:34:04.769605 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786401215402979-17317-0/minicluster-data/ts-2/wal/instance
I20260810 22:34:04.769912 17317 external_mini_cluster.cc:1428] Setting key 8120e4a39b3ea37928dbd8b940df218a
I20260810 22:34:04.770503 22913 heartbeater.cc:347] Connected to a master server at 127.16.233.126:42743
I20260810 22:34:04.770769 22913 heartbeater.cc:464] Registering TS with master...
I20260810 22:34:04.771309 22913 heartbeater.cc:511] Master 127.16.233.126:42743 requested a full tablet report, sending...
I20260810 22:34:04.772425 22424 ts_manager.cc:194] Registered new tserver with Master: 23ff9fdd63ae4a48b56ddabfae058e03 (127.16.233.67:33533)
I20260810 22:34:04.784845 17317 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 22:34:04.789477 22469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:04.787299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36284 (local address 127.16.233.126:42743)
0810 22:34:04.787492 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:04.787496 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:04.787670 (+ 174us) server_negotiation.cc:407] Connection header received
0810 22:34:04.787785 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:04.787788 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:04.787898 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:04.787986 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:04.789022 (+ 1036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:04.789026 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:04.789029 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:34:04.789042 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:04.789065 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:34:04.789074 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:04.789167 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:04.789299 (+ 132us) server_negotiation.cc:299] Negotiation successful
0810 22:34:04.789352 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260810 22:34:04.793475 22424 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36284:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 22:34:04.796636 22424 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:34:04.810153 22929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:04.807508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.65:38099 (local address 127.0.0.1:55560)
0810 22:34:04.807829 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:04.807844 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:04.807913 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:04.808657 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:04.808661 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:04.808677 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:04.809352 (+ 675us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 22:34:04.809354 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:04.809376 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 22:34:04.809449 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:34:04.809750 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:34:04.809760 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:34:04.809766 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 22:34:04.809917 (+ 151us) client_negotiation.cc:240] Negotiation successful
0810 22:34:04.809997 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":92,"threads_started":1}
I20260810 22:34:04.810166 22930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:04.807789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55560 (local address 127.16.233.65:38099)
0810 22:34:04.808225 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:04.808237 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:04.808263 (+ 26us) server_negotiation.cc:407] Connection header received
0810 22:34:04.808326 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:04.808331 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:04.808503 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:04.808913 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:04.809561 (+ 648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:04.809568 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:04.809580 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:34:04.809590 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:04.809636 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:34:04.809666 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:04.809774 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:04.809932 (+ 158us) server_negotiation.cc:299] Negotiation successful
0810 22:34:04.810010 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":98,"threads_started":1}
I20260810 22:34:04.810356 22925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:04.806610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.67:33533 (local address 127.0.0.1:60856)
0810 22:34:04.807169 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:04.807225 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:04.807357 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:04.808720 (+ 1363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:04.808723 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:04.808738 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:04.809558 (+ 820us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 22:34:04.809563 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:04.809578 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 22:34:04.809610 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:34:04.810093 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:34:04.810096 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:34:04.810101 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 22:34:04.810165 (+ 64us) client_negotiation.cc:240] Negotiation successful
0810 22:34:04.810218 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":114,"threads_started":1}
I20260810 22:34:04.810766 22927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:04.806933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.66:37151 (local address 127.0.0.1:47402)
0810 22:34:04.807562 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:04.807578 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:04.807651 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:04.808463 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:04.808470 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:04.808553 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:04.809556 (+ 1003us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 22:34:04.809559 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:04.809570 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 22:34:04.809590 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:34:04.810110 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:34:04.810113 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:34:04.810115 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 22:34:04.810180 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 22:34:04.810641 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"mutex_wait_us":422,"thread_start_us":95,"threads_started":1}
I20260810 22:34:04.811026 22926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:04.806819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60856 (local address 127.16.233.67:33533)
0810 22:34:04.808310 (+ 1491us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:04.808326 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:04.808351 (+ 25us) server_negotiation.cc:407] Connection header received
0810 22:34:04.808404 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:04.808410 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:04.808565 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:04.809022 (+ 457us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:04.809900 (+ 878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:04.809905 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:04.809914 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:34:04.809925 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:04.809957 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:34:04.809981 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:04.810726 (+ 745us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:04.810776 (+ 50us) server_negotiation.cc:299] Negotiation successful
0810 22:34:04.810869 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1388,"thread_start_us":126,"threads_started":1}
I20260810 22:34:04.811115 22928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:04.807342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47402 (local address 127.16.233.66:37151)
0810 22:34:04.807935 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:04.807951 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:04.807981 (+ 30us) server_negotiation.cc:407] Connection header received
0810 22:34:04.808043 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:04.808049 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:04.808265 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:04.808447 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:04.809861 (+ 1414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:04.809866 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:04.809877 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:34:04.809888 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:04.809925 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:34:04.809951 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:04.810072 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:04.810248 (+ 176us) server_negotiation.cc:299] Negotiation successful
0810 22:34:04.810328 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":164,"threads_started":1}
I20260810 22:34:04.814005 22559 tablet_service.cc:1467] Processing CreateTablet for tablet 6a312175c4f545fda8fe64cf6c5ad753 (DEFAULT_TABLE table=test-table [id=ccd281a1303c4d4fa445aa04fd5bc63f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:34:04.814005 22840 tablet_service.cc:1467] Processing CreateTablet for tablet 6a312175c4f545fda8fe64cf6c5ad753 (DEFAULT_TABLE table=test-table [id=ccd281a1303c4d4fa445aa04fd5bc63f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:34:04.814607 22698 tablet_service.cc:1467] Processing CreateTablet for tablet 6a312175c4f545fda8fe64cf6c5ad753 (DEFAULT_TABLE table=test-table [id=ccd281a1303c4d4fa445aa04fd5bc63f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:34:04.815263 22840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a312175c4f545fda8fe64cf6c5ad753. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:34:04.815263 22559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a312175c4f545fda8fe64cf6c5ad753. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:34:04.815670 22698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a312175c4f545fda8fe64cf6c5ad753. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:34:04.822475 22931 tablet_bootstrap.cc:492] T 6a312175c4f545fda8fe64cf6c5ad753 P 6017e08b7db64e29bd259c112d69e2dd: Bootstrap starting.
I20260810 22:34:04.824008 22933 tablet_bootstrap.cc:492] T 6a312175c4f545fda8fe64cf6c5ad753 P 23ff9fdd63ae4a48b56ddabfae058e03: Bootstrap starting.
I20260810 22:34:04.824075 22932 tablet_bootstrap.cc:492] T 6a312175c4f545fda8fe64cf6c5ad753 P 1d284c2b05824f99a2ec002051e7fc30: Bootstrap starting.
I20260810 22:34:04.825697 22931 tablet_bootstrap.cc:654] T 6a312175c4f545fda8fe64cf6c5ad753 P 6017e08b7db64e29bd259c112d69e2dd: Neither blocks nor log segments found. Creating new log.
I20260810 22:34:04.826558 22932 tablet_bootstrap.cc:654] T 6a312175c4f545fda8fe64cf6c5ad753 P 1d284c2b05824f99a2ec002051e7fc30: Neither blocks nor log segments found. Creating new log.
I20260810 22:34:04.826560 22933 tablet_bootstrap.cc:654] T 6a312175c4f545fda8fe64cf6c5ad753 P 23ff9fdd63ae4a48b56ddabfae058e03: Neither blocks nor log segments found. Creating new log.
I20260810 22:34:04.826596 22931 log.cc:824] T 6a312175c4f545fda8fe64cf6c5ad753 P 6017e08b7db64e29bd259c112d69e2dd: Log is configured to *not* fsync() on all Append() calls
I20260810 22:34:04.827466 22932 log.cc:824] T 6a312175c4f545fda8fe64cf6c5ad753 P 1d284c2b05824f99a2ec002051e7fc30: Log is configured to *not* fsync() on all Append() calls
I20260810 22:34:04.827466 22933 log.cc:824] T 6a312175c4f545fda8fe64cf6c5ad753 P 23ff9fdd63ae4a48b56ddabfae058e03: Log is configured to *not* fsync() on all Append() calls
I20260810 22:34:04.829098 22931 tablet_bootstrap.cc:492] T 6a312175c4f545fda8fe64cf6c5ad753 P 6017e08b7db64e29bd259c112d69e2dd: No bootstrap required, opened a new log
I20260810 22:34:04.829340 22931 ts_tablet_manager.cc:1397] T 6a312175c4f545fda8fe64cf6c5ad753 P 6017e08b7db64e29bd259c112d69e2dd: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260810 22:34:04.829483 22933 tablet_bootstrap.cc:492] T 6a312175c4f545fda8fe64cf6c5ad753 P 23ff9fdd63ae4a48b56ddabfae058e03: No bootstrap required, opened a new log
I20260810 22:34:04.829685 22933 ts_tablet_manager.cc:1397] T 6a312175c4f545fda8fe64cf6c5ad753 P 23ff9fdd63ae4a48b56ddabfae058e03: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 22:34:04.829573 22932 tablet_bootstrap.cc:492] T 6a312175c4f545fda8fe64cf6c5ad753 P 1d284c2b05824f99a2ec002051e7fc30: No bootstrap required, opened a new log
I20260810 22:34:04.830094 22932 ts_tablet_manager.cc:1397] T 6a312175c4f545fda8fe64cf6c5ad753 P 1d284c2b05824f99a2ec002051e7fc30: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 22:34:04.833585 22933 raft_consensus.cc:348] T 6a312175c4f545fda8fe64cf6c5ad753 P 23ff9fdd63ae4a48b56ddabfae058e03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ff9fdd63ae4a48b56ddabfae058e03" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 33533 } } peers { permanent_uuid: "6017e08b7db64e29bd259c112d69e2dd" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 37151 } } peers { permanent_uuid: "1d284c2b05824f99a2ec002051e7fc30" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 38099 } }
I20260810 22:34:04.833900 22933 raft_consensus.cc:374] T 6a312175c4f545fda8fe64cf6c5ad753 P 23ff9fdd63ae4a48b56ddabfae058e03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:34:04.833984 22933 raft_consensus.cc:729] T 6a312175c4f545fda8fe64cf6c5ad753 P 23ff9fdd63ae4a48b56ddabfae058e03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23ff9fdd63ae4a48b56ddabfae058e03, State: Initialized, Role: FOLLOWER
I20260810 22:34:04.834246 22931 raft_consensus.cc:348] T 6a312175c4f545fda8fe64cf6c5ad753 P 6017e08b7db64e29bd259c112d69e2dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ff9fdd63ae4a48b56ddabfae058e03" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 33533 } } peers { permanent_uuid: "6017e08b7db64e29bd259c112d69e2dd" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 37151 } } peers { permanent_uuid: "1d284c2b05824f99a2ec002051e7fc30" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 38099 } }
I20260810 22:34:04.834573 22931 raft_consensus.cc:374] T 6a312175c4f545fda8fe64cf6c5ad753 P 6017e08b7db64e29bd259c112d69e2dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:34:04.834518 22933 consensus_queue.cc:260] T 6a312175c4f545fda8fe64cf6c5ad753 P 23ff9fdd63ae4a48b56ddabfae058e03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ff9fdd63ae4a48b56ddabfae058e03" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 33533 } } peers { permanent_uuid: "6017e08b7db64e29bd259c112d69e2dd" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 37151 } } peers { permanent_uuid: "1d284c2b05824f99a2ec002051e7fc30" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 38099 } }
I20260810 22:34:04.834677 22931 raft_consensus.cc:729] T 6a312175c4f545fda8fe64cf6c5ad753 P 6017e08b7db64e29bd259c112d69e2dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6017e08b7db64e29bd259c112d69e2dd, State: Initialized, Role: FOLLOWER
I20260810 22:34:04.835151 22931 consensus_queue.cc:260] T 6a312175c4f545fda8fe64cf6c5ad753 P 6017e08b7db64e29bd259c112d69e2dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ff9fdd63ae4a48b56ddabfae058e03" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 33533 } } peers { permanent_uuid: "6017e08b7db64e29bd259c112d69e2dd" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 37151 } } peers { permanent_uuid: "1d284c2b05824f99a2ec002051e7fc30" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 38099 } }
I20260810 22:34:04.835413 22913 heartbeater.cc:503] Master 127.16.233.126:42743 was elected leader, sending a full tablet report...
I20260810 22:34:04.835549 22932 raft_consensus.cc:348] T 6a312175c4f545fda8fe64cf6c5ad753 P 1d284c2b05824f99a2ec002051e7fc30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ff9fdd63ae4a48b56ddabfae058e03" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 33533 } } peers { permanent_uuid: "6017e08b7db64e29bd259c112d69e2dd" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 37151 } } peers { permanent_uuid: "1d284c2b05824f99a2ec002051e7fc30" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 38099 } }
I20260810 22:34:04.835819 22932 raft_consensus.cc:374] T 6a312175c4f545fda8fe64cf6c5ad753 P 1d284c2b05824f99a2ec002051e7fc30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:34:04.835904 22932 raft_consensus.cc:729] T 6a312175c4f545fda8fe64cf6c5ad753 P 1d284c2b05824f99a2ec002051e7fc30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d284c2b05824f99a2ec002051e7fc30, State: Initialized, Role: FOLLOWER
I20260810 22:34:04.836398 22931 ts_tablet_manager.cc:1428] T 6a312175c4f545fda8fe64cf6c5ad753 P 6017e08b7db64e29bd259c112d69e2dd: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
I20260810 22:34:04.836349 22932 consensus_queue.cc:260] T 6a312175c4f545fda8fe64cf6c5ad753 P 1d284c2b05824f99a2ec002051e7fc30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ff9fdd63ae4a48b56ddabfae058e03" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 33533 } } peers { permanent_uuid: "6017e08b7db64e29bd259c112d69e2dd" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 37151 } } peers { permanent_uuid: "1d284c2b05824f99a2ec002051e7fc30" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 38099 } }
I20260810 22:34:04.836802 22772 heartbeater.cc:503] Master 127.16.233.126:42743 was elected leader, sending a full tablet report...
I20260810 22:34:04.836907 22933 ts_tablet_manager.cc:1428] T 6a312175c4f545fda8fe64cf6c5ad753 P 23ff9fdd63ae4a48b56ddabfae058e03: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260810 22:34:04.837518 22631 heartbeater.cc:503] Master 127.16.233.126:42743 was elected leader, sending a full tablet report...
I20260810 22:34:04.839243 22932 ts_tablet_manager.cc:1428] T 6a312175c4f545fda8fe64cf6c5ad753 P 1d284c2b05824f99a2ec002051e7fc30: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260810 22:34:04.851827 22938 raft_consensus.cc:482] T 6a312175c4f545fda8fe64cf6c5ad753 P 6017e08b7db64e29bd259c112d69e2dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:34:04.852113 22938 raft_consensus.cc:504] T 6a312175c4f545fda8fe64cf6c5ad753 P 6017e08b7db64e29bd259c112d69e2dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ff9fdd63ae4a48b56ddabfae058e03" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 33533 } } peers { permanent_uuid: "6017e08b7db64e29bd259c112d69e2dd" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 37151 } } peers { permanent_uuid: "1d284c2b05824f99a2ec002051e7fc30" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 38099 } }
I20260810 22:34:04.853726 22938 leader_election.cc:288] T 6a312175c4f545fda8fe64cf6c5ad753 P 6017e08b7db64e29bd259c112d69e2dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23ff9fdd63ae4a48b56ddabfae058e03 (127.16.233.67:33533), 1d284c2b05824f99a2ec002051e7fc30 (127.16.233.65:38099)
I20260810 22:34:04.856264 22930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:04.854108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:37281 (local address 127.16.233.65:38099)
0810 22:34:04.854254 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:04.854259 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:04.854538 (+ 279us) server_negotiation.cc:407] Connection header received
0810 22:34:04.854601 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:04.854607 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:04.854656 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:04.854775 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:04.855716 (+ 941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:04.855720 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:04.855805 (+ 85us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:34:04.855825 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:04.855850 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:34:04.855863 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:04.855990 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:04.856089 (+ 99us) server_negotiation.cc:299] Negotiation successful
0810 22:34:04.856129 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 22:34:04.856304 22926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:04.854240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:56121 (local address 127.16.233.67:33533)
0810 22:34:04.854404 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:04.854417 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:04.854433 (+ 16us) server_negotiation.cc:407] Connection header received
0810 22:34:04.854479 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:04.854511 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:04.854559 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:04.854665 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:04.855717 (+ 1052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:04.855720 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:04.855723 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:34:04.855738 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:04.855764 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:34:04.855776 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:04.855900 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:04.856068 (+ 168us) server_negotiation.cc:299] Negotiation successful
0810 22:34:04.856120 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 22:34:04.856264 22775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:04.853967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.67:33533 (local address 127.16.233.66:56121)
0810 22:34:04.854152 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:04.854174 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:04.854283 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:04.854690 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:04.854692 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:04.854705 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:04.855490 (+ 785us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 22:34:04.855492 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:04.855501 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 22:34:04.855534 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:34:04.855889 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:34:04.855892 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:34:04.855895 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 22:34:04.856028 (+ 133us) client_negotiation.cc:240] Negotiation successful
0810 22:34:04.856080 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260810 22:34:04.856290 22940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:04.854014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.65:38099 (local address 127.16.233.66:37281)
0810 22:34:04.854425 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:04.854436 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:04.854520 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:04.854783 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:04.854786 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:04.854796 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:04.855476 (+ 680us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 22:34:04.855479 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:04.855495 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 22:34:04.855534 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:34:04.855973 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:34:04.855979 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:34:04.855984 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 22:34:04.856078 (+ 94us) client_negotiation.cc:240] Negotiation successful
0810 22:34:04.856109 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":168,"threads_started":1}
I20260810 22:34:04.857160 22861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a312175c4f545fda8fe64cf6c5ad753" candidate_uuid: "6017e08b7db64e29bd259c112d69e2dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ff9fdd63ae4a48b56ddabfae058e03" is_pre_election: true
I20260810 22:34:04.857160 22580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a312175c4f545fda8fe64cf6c5ad753" candidate_uuid: "6017e08b7db64e29bd259c112d69e2dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d284c2b05824f99a2ec002051e7fc30" is_pre_election: true
I20260810 22:34:04.857513 22580 raft_consensus.cc:2454] T 6a312175c4f545fda8fe64cf6c5ad753 P 1d284c2b05824f99a2ec002051e7fc30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6017e08b7db64e29bd259c112d69e2dd in term 0.
I20260810 22:34:04.857513 22861 raft_consensus.cc:2454] T 6a312175c4f545fda8fe64cf6c5ad753 P 23ff9fdd63ae4a48b56ddabfae058e03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6017e08b7db64e29bd259c112d69e2dd in term 0.
I20260810 22:34:04.858146 22652 leader_election.cc:302] T 6a312175c4f545fda8fe64cf6c5ad753 P 6017e08b7db64e29bd259c112d69e2dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1d284c2b05824f99a2ec002051e7fc30, 23ff9fdd63ae4a48b56ddabfae058e03, 6017e08b7db64e29bd259c112d69e2dd; no voters:
I20260810 22:34:04.858485 22938 raft_consensus.cc:2781] T 6a312175c4f545fda8fe64cf6c5ad753 P 6017e08b7db64e29bd259c112d69e2dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:34:04.858585 22938 raft_consensus.cc:482] T 6a312175c4f545fda8fe64cf6c5ad753 P 6017e08b7db64e29bd259c112d69e2dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:34:04.858637 22938 raft_consensus.cc:3037] T 6a312175c4f545fda8fe64cf6c5ad753 P 6017e08b7db64e29bd259c112d69e2dd [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:34:04.859764 22938 raft_consensus.cc:504] T 6a312175c4f545fda8fe64cf6c5ad753 P 6017e08b7db64e29bd259c112d69e2dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ff9fdd63ae4a48b56ddabfae058e03" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 33533 } } peers { permanent_uuid: "6017e08b7db64e29bd259c112d69e2dd" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 37151 } } peers { permanent_uuid: "1d284c2b05824f99a2ec002051e7fc30" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 38099 } }
I20260810 22:34:04.860217 22938 leader_election.cc:288] T 6a312175c4f545fda8fe64cf6c5ad753 P 6017e08b7db64e29bd259c112d69e2dd [CANDIDATE]: Term 1 election: Requested vote from peers 23ff9fdd63ae4a48b56ddabfae058e03 (127.16.233.67:33533), 1d284c2b05824f99a2ec002051e7fc30 (127.16.233.65:38099)
I20260810 22:34:04.860695 22861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a312175c4f545fda8fe64cf6c5ad753" candidate_uuid: "6017e08b7db64e29bd259c112d69e2dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23ff9fdd63ae4a48b56ddabfae058e03"
I20260810 22:34:04.860737 22580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a312175c4f545fda8fe64cf6c5ad753" candidate_uuid: "6017e08b7db64e29bd259c112d69e2dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d284c2b05824f99a2ec002051e7fc30"
I20260810 22:34:04.860875 22861 raft_consensus.cc:3037] T 6a312175c4f545fda8fe64cf6c5ad753 P 23ff9fdd63ae4a48b56ddabfae058e03 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:34:04.860967 22580 raft_consensus.cc:3037] T 6a312175c4f545fda8fe64cf6c5ad753 P 1d284c2b05824f99a2ec002051e7fc30 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:34:04.862001 22861 raft_consensus.cc:2454] T 6a312175c4f545fda8fe64cf6c5ad753 P 23ff9fdd63ae4a48b56ddabfae058e03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6017e08b7db64e29bd259c112d69e2dd in term 1.
I20260810 22:34:04.862200 22580 raft_consensus.cc:2454] T 6a312175c4f545fda8fe64cf6c5ad753 P 1d284c2b05824f99a2ec002051e7fc30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6017e08b7db64e29bd259c112d69e2dd in term 1.
I20260810 22:34:04.862403 22652 leader_election.cc:302] T 6a312175c4f545fda8fe64cf6c5ad753 P 6017e08b7db64e29bd259c112d69e2dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23ff9fdd63ae4a48b56ddabfae058e03, 6017e08b7db64e29bd259c112d69e2dd; no voters:
I20260810 22:34:04.862591 22938 raft_consensus.cc:2781] T 6a312175c4f545fda8fe64cf6c5ad753 P 6017e08b7db64e29bd259c112d69e2dd [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:34:04.862917 22938 raft_consensus.cc:686] T 6a312175c4f545fda8fe64cf6c5ad753 P 6017e08b7db64e29bd259c112d69e2dd [term 1 LEADER]: Becoming Leader. State: Replica: 6017e08b7db64e29bd259c112d69e2dd, State: Running, Role: LEADER
I20260810 22:34:04.863284 22938 consensus_queue.cc:237] T 6a312175c4f545fda8fe64cf6c5ad753 P 6017e08b7db64e29bd259c112d69e2dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ff9fdd63ae4a48b56ddabfae058e03" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 33533 } } peers { permanent_uuid: "6017e08b7db64e29bd259c112d69e2dd" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 37151 } } peers { permanent_uuid: "1d284c2b05824f99a2ec002051e7fc30" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 38099 } }
W20260810 22:34:04.866797 22632 tablet.cc:2369] T 6a312175c4f545fda8fe64cf6c5ad753 P 1d284c2b05824f99a2ec002051e7fc30: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:34:04.867296 22423 catalog_manager.cc:5495] T 6a312175c4f545fda8fe64cf6c5ad753 P 6017e08b7db64e29bd259c112d69e2dd reported cstate change: term changed from 0 to 1, leader changed from <none> to 6017e08b7db64e29bd259c112d69e2dd (127.16.233.66). New cstate: current_term: 1 leader_uuid: "6017e08b7db64e29bd259c112d69e2dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23ff9fdd63ae4a48b56ddabfae058e03" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 33533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6017e08b7db64e29bd259c112d69e2dd" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 37151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d284c2b05824f99a2ec002051e7fc30" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 38099 } health_report { overall_health: UNKNOWN } } }
I20260810 22:34:04.878819 22928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:04.877214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47412 (local address 127.16.233.66:37151)
0810 22:34:04.877359 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:04.877363 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:04.877378 (+ 15us) server_negotiation.cc:407] Connection header received
0810 22:34:04.877493 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:04.877496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:04.877558 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:04.877638 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:04.878445 (+ 807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:04.878449 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:04.878451 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:34:04.878463 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:04.878483 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:34:04.878491 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:04.878573 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:04.878660 (+ 87us) server_negotiation.cc:299] Negotiation successful
0810 22:34:04.878694 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 22:34:04.885576 22861 raft_consensus.cc:1261] T 6a312175c4f545fda8fe64cf6c5ad753 P 23ff9fdd63ae4a48b56ddabfae058e03 [term 1 FOLLOWER]: Refusing update from remote peer 6017e08b7db64e29bd259c112d69e2dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:34:04.885558 22580 raft_consensus.cc:1261] T 6a312175c4f545fda8fe64cf6c5ad753 P 1d284c2b05824f99a2ec002051e7fc30 [term 1 FOLLOWER]: Refusing update from remote peer 6017e08b7db64e29bd259c112d69e2dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:34:04.886416 22938 consensus_queue.cc:1035] T 6a312175c4f545fda8fe64cf6c5ad753 P 6017e08b7db64e29bd259c112d69e2dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "23ff9fdd63ae4a48b56ddabfae058e03" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 33533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:34:04.886713 22941 consensus_queue.cc:1035] T 6a312175c4f545fda8fe64cf6c5ad753 P 6017e08b7db64e29bd259c112d69e2dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d284c2b05824f99a2ec002051e7fc30" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 38099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:34:04.895351 22944 mvcc.cc:205] Tried to move back new op lower bound from 7317099499042119680 to 7317099498963111936. Current Snapshot: MvccSnapshot[applied={T|T < 7317099499042119680}]
I20260810 22:34:04.897136 22945 mvcc.cc:205] Tried to move back new op lower bound from 7317099499042119680 to 7317099498963111936. Current Snapshot: MvccSnapshot[applied={T|T < 7317099499042119680}]
I20260810 22:34:04.897667 22946 mvcc.cc:205] Tried to move back new op lower bound from 7317099499042119680 to 7317099498963111936. Current Snapshot: MvccSnapshot[applied={T|T < 7317099499042119680}]
I20260810 22:34:04.906998 22421 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:34:04.907222 22421 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:34:04.910138 22421 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2e69d3ea627342978b7e809974aa55da: Sending DeleteTablet for 3 replicas of tablet 6a312175c4f545fda8fe64cf6c5ad753
I20260810 22:34:04.911088 22840 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a312175c4f545fda8fe64cf6c5ad753 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:34:04 UTC) from {username='slave'} at 127.0.0.1:60856
I20260810 22:34:04.911134 22698 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a312175c4f545fda8fe64cf6c5ad753 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:34:04 UTC) from {username='slave'} at 127.0.0.1:47402
I20260810 22:34:04.911196 22559 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a312175c4f545fda8fe64cf6c5ad753 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:34:04 UTC) from {username='slave'} at 127.0.0.1:55560
I20260810 22:34:04.911764 22954 tablet_replica.cc:330] T 6a312175c4f545fda8fe64cf6c5ad753 P 6017e08b7db64e29bd259c112d69e2dd: stopping tablet replica
I20260810 22:34:04.911928 22955 tablet_replica.cc:330] T 6a312175c4f545fda8fe64cf6c5ad753 P 23ff9fdd63ae4a48b56ddabfae058e03: stopping tablet replica
I20260810 22:34:04.912146 22954 raft_consensus.cc:2229] T 6a312175c4f545fda8fe64cf6c5ad753 P 6017e08b7db64e29bd259c112d69e2dd [term 1 LEADER]: Raft consensus shutting down.
I20260810 22:34:04.912254 22955 raft_consensus.cc:2229] T 6a312175c4f545fda8fe64cf6c5ad753 P 23ff9fdd63ae4a48b56ddabfae058e03 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:34:04.912539 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 22491
I20260810 22:34:04.912606 22955 raft_consensus.cc:2258] T 6a312175c4f545fda8fe64cf6c5ad753 P 23ff9fdd63ae4a48b56ddabfae058e03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:34:04.912609 22954 raft_consensus.cc:2258] T 6a312175c4f545fda8fe64cf6c5ad753 P 6017e08b7db64e29bd259c112d69e2dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:34:04.915181 22954 ts_tablet_manager.cc:1905] T 6a312175c4f545fda8fe64cf6c5ad753 P 6017e08b7db64e29bd259c112d69e2dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:34:04.919231 22954 ts_tablet_manager.cc:1918] T 6a312175c4f545fda8fe64cf6c5ad753 P 6017e08b7db64e29bd259c112d69e2dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:34:04.919382 22954 log.cc:1196] T 6a312175c4f545fda8fe64cf6c5ad753 P 6017e08b7db64e29bd259c112d69e2dd: Deleting WAL directory at /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786401215402979-17317-0/minicluster-data/ts-1/wal/wals/6a312175c4f545fda8fe64cf6c5ad753
I20260810 22:34:04.919795 22954 ts_tablet_manager.cc:1939] T 6a312175c4f545fda8fe64cf6c5ad753 P 6017e08b7db64e29bd259c112d69e2dd: Deleting consensus metadata
I20260810 22:34:04.919941 22955 ts_tablet_manager.cc:1905] T 6a312175c4f545fda8fe64cf6c5ad753 P 23ff9fdd63ae4a48b56ddabfae058e03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:34:04.921382 22408 catalog_manager.cc:4841] TS 6017e08b7db64e29bd259c112d69e2dd (127.16.233.66:37151): tablet 6a312175c4f545fda8fe64cf6c5ad753 (table test-table [id=ccd281a1303c4d4fa445aa04fd5bc63f]) successfully deleted
W20260810 22:34:04.923307 22407 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.233.65:38099 (error 108)
I20260810 22:34:04.923381 22955 ts_tablet_manager.cc:1918] T 6a312175c4f545fda8fe64cf6c5ad753 P 23ff9fdd63ae4a48b56ddabfae058e03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:34:04.923496 22955 log.cc:1196] T 6a312175c4f545fda8fe64cf6c5ad753 P 23ff9fdd63ae4a48b56ddabfae058e03: Deleting WAL directory at /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786401215402979-17317-0/minicluster-data/ts-2/wal/wals/6a312175c4f545fda8fe64cf6c5ad753
I20260810 22:34:04.923925 22955 ts_tablet_manager.cc:1939] T 6a312175c4f545fda8fe64cf6c5ad753 P 23ff9fdd63ae4a48b56ddabfae058e03: Deleting consensus metadata
I20260810 22:34:04.924250 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 22634
I20260810 22:34:04.925812 22927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:04.924443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.65:38099 (local address 127.0.0.1:55566)
0810 22:34:04.925598 (+ 1155us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:04.925690 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.65:38099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1036}
W20260810 22:34:04.926301 22407 catalog_manager.cc:4584] TS 1d284c2b05824f99a2ec002051e7fc30 (127.16.233.65:38099): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6a312175c4f545fda8fe64cf6c5ad753: Network error: Client connection negotiation failed: client connection to 127.16.233.65:38099: connect: Connection refused (error 111)
I20260810 22:34:04.928172 22411 catalog_manager.cc:4841] TS 23ff9fdd63ae4a48b56ddabfae058e03 (127.16.233.67:33533): tablet 6a312175c4f545fda8fe64cf6c5ad753 (table test-table [id=ccd281a1303c4d4fa445aa04fd5bc63f]) successfully deleted
I20260810 22:34:04.933912 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 22776
I20260810 22:34:04.942689 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 22392
2026-08-10T22:34:04Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1067 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260810 22:34:04.968875 17317 mini_oidc.cc:152] Starting JWKS server
I20260810 22:34:04.970772 17317 webserver.cc:460] Webserver started at https://127.0.0.1:34501/ using document root <none> and password file <none>
I20260810 22:34:04.970978 17317 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 22:34:04.971293 17317 webserver.cc:460] Webserver started at http://127.0.0.1:36027/ using document root <none> and password file <none>
2026-08-10T22:34:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:34:04Z Disabled control of system clock
I20260810 22:34:04.983345 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786401215402979-17317-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786401215402979-17317-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786401215402979-17317-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:41179
--webserver_interface=127.16.233.126
--webserver_port=0
--builtin_ntp_servers=127.16.233.84:38905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.126:41179
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34501/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786401215402979-17317-0/testchainca.cert with env {}
W20260810 22:34:05.103121 22963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:34:05.103446 22963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:34:05.103492 22963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:34:05.107879 22963 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:34:05.107970 22963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:34:05.107986 22963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:34:05.108004 22963 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:34:05.108021 22963 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 22:34:05.108037 22963 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 22:34:05.108054 22963 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34501/jwks.json
I20260810 22:34:05.114135 22963 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:38905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786401215402979-17317-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786401215402979-17317-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.126:41179
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:41179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34501/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786401215402979-17317-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786401215402979-17317-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:34:05.115343 22963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:34:05.116400 22963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:34:05.125941 22968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:05.125962 22970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:05.126011 22973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:05.126101 22963 server_base.cc:1034] running on GCE node
I20260810 22:34:05.127148 22963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:34:05.128362 22963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:34:05.129534 22963 hybrid_clock.cc:648] HybridClock initialized: now 1786401245129441 us; error 108 us; skew 500 ppm
I20260810 22:34:05.131611 22963 webserver.cc:460] Webserver started at http://127.16.233.126:40733/ using document root <none> and password file <none>
I20260810 22:34:05.132194 22963 fs_manager.cc:359] Metadata directory not provided
I20260810 22:34:05.132261 22963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:34:05.132439 22963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:34:05.134769 22963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786401215402979-17317-0/minicluster-data/master-0/data/instance:
uuid: "a5f7126115b94a56a64104b9cbd76e8d"
format_stamp: "Formatted at 2026-08-10 22:34:05 on dist-test-slave-77v9"
server_key: "197b7ccf48cc4ebd0fedb2fdcf74ef6e"
server_key_iv: "6365c4fd696f342b2eeedc8f57aa7fc9"
server_key_version: "encryptionkey@0"
I20260810 22:34:05.135416 22963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786401215402979-17317-0/minicluster-data/master-0/wal/instance:
uuid: "a5f7126115b94a56a64104b9cbd76e8d"
format_stamp: "Formatted at 2026-08-10 22:34:05 on dist-test-slave-77v9"
server_key: "197b7ccf48cc4ebd0fedb2fdcf74ef6e"
server_key_iv: "6365c4fd696f342b2eeedc8f57aa7fc9"
server_key_version: "encryptionkey@0"
I20260810 22:34:05.139853 22963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260810 22:34:05.142768 22979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:05.144469 22963 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 22:34:05.144658 22963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786401215402979-17317-0/minicluster-data/master-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786401215402979-17317-0/minicluster-data/master-0/wal
uuid: "a5f7126115b94a56a64104b9cbd76e8d"
format_stamp: "Formatted at 2026-08-10 22:34:05 on dist-test-slave-77v9"
server_key: "197b7ccf48cc4ebd0fedb2fdcf74ef6e"
server_key_iv: "6365c4fd696f342b2eeedc8f57aa7fc9"
server_key_version: "encryptionkey@0"
I20260810 22:34:05.144876 22963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786401215402979-17317-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786401215402979-17317-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786401215402979-17317-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:34:05.158608 22963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:34:05.159441 22963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:34:05.159680 22963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:34:05.168634 22963 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.126:41179
I20260810 22:34:05.168639 23041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.126:41179 every 8 connection(s)
I20260810 22:34:05.169819 22963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
I20260810 22:34:05.171164 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 22963
I20260810 22:34:05.171339 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786401215402979-17317-0/minicluster-data/master-0/wal/instance
I20260810 22:34:05.171686 17317 external_mini_cluster.cc:1428] Setting key 335156e562e6649725c798d7e55ec544
I20260810 22:34:05.174376 23042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:34:05.181185 23042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5f7126115b94a56a64104b9cbd76e8d: Bootstrap starting.
I20260810 22:34:05.181838 23045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:05.173398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40878 (local address 127.16.233.126:41179)
0810 22:34:05.174158 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:05.174183 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:05.174222 (+ 39us) server_negotiation.cc:407] Connection header received
0810 22:34:05.174990 (+ 768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:05.175009 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:05.175430 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:05.175796 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:05.176240 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:05.177340 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:05.178249 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:05.178742 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:05.178945 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:05.179932 (+ 987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:05.179938 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:05.179951 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:34:05.180351 (+ 400us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:05.180411 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:34:05.180432 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:05.180546 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:05.180703 (+ 157us) server_negotiation.cc:299] Negotiation successful
0810 22:34:05.181028 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":151,"threads_started":1}
I20260810 22:34:05.183981 23042 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5f7126115b94a56a64104b9cbd76e8d: Neither blocks nor log segments found. Creating new log.
I20260810 22:34:05.184883 17317 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 22:34:05.185003 23042 log.cc:824] T 00000000000000000000000000000000 P a5f7126115b94a56a64104b9cbd76e8d: Log is configured to *not* fsync() on all Append() calls
I20260810 22:34:05.187357 23042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5f7126115b94a56a64104b9cbd76e8d: No bootstrap required, opened a new log
I20260810 22:34:05.191491 23042 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5f7126115b94a56a64104b9cbd76e8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5f7126115b94a56a64104b9cbd76e8d" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 41179 } }
I20260810 22:34:05.191781 23042 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5f7126115b94a56a64104b9cbd76e8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:34:05.191883 23042 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5f7126115b94a56a64104b9cbd76e8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5f7126115b94a56a64104b9cbd76e8d, State: Initialized, Role: FOLLOWER
I20260810 22:34:05.192368 23042 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5f7126115b94a56a64104b9cbd76e8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5f7126115b94a56a64104b9cbd76e8d" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 41179 } }
I20260810 22:34:05.192513 23042 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5f7126115b94a56a64104b9cbd76e8d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:34:05.192610 23042 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5f7126115b94a56a64104b9cbd76e8d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:34:05.192762 23042 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5f7126115b94a56a64104b9cbd76e8d [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:34:05.194006 23042 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5f7126115b94a56a64104b9cbd76e8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5f7126115b94a56a64104b9cbd76e8d" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 41179 } }
I20260810 22:34:05.194461 23042 leader_election.cc:302] T 00000000000000000000000000000000 P a5f7126115b94a56a64104b9cbd76e8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5f7126115b94a56a64104b9cbd76e8d; no voters:
I20260810 22:34:05.194870 23042 leader_election.cc:288] T 00000000000000000000000000000000 P a5f7126115b94a56a64104b9cbd76e8d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:34:05.195235 23047 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5f7126115b94a56a64104b9cbd76e8d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:34:05.195577 23047 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5f7126115b94a56a64104b9cbd76e8d [term 1 LEADER]: Becoming Leader. State: Replica: a5f7126115b94a56a64104b9cbd76e8d, State: Running, Role: LEADER
I20260810 22:34:05.195955 23047 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5f7126115b94a56a64104b9cbd76e8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5f7126115b94a56a64104b9cbd76e8d" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 41179 } }
I20260810 22:34:05.196424 23042 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5f7126115b94a56a64104b9cbd76e8d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:34:05.197968 23049 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5f7126115b94a56a64104b9cbd76e8d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5f7126115b94a56a64104b9cbd76e8d. Latest consensus state: current_term: 1 leader_uuid: "a5f7126115b94a56a64104b9cbd76e8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5f7126115b94a56a64104b9cbd76e8d" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 41179 } } }
I20260810 22:34:05.198136 23049 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5f7126115b94a56a64104b9cbd76e8d [sys.catalog]: This master's current role is: LEADER
I20260810 22:34:05.198616 23058 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:34:05.197978 23048 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5f7126115b94a56a64104b9cbd76e8d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a5f7126115b94a56a64104b9cbd76e8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5f7126115b94a56a64104b9cbd76e8d" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 41179 } } }
I20260810 22:34:05.199163 23048 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5f7126115b94a56a64104b9cbd76e8d [sys.catalog]: This master's current role is: LEADER
I20260810 22:34:05.202008 23058 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:34:05.209249 23058 catalog_manager.cc:1347] Generated new cluster ID: 0a71659bcf4a43d6a9b14b284cab27e0
I20260810 22:34:05.209378 23058 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:34:05.234182 23058 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:34:05.235306 23058 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:34:05.244627 23058 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a5f7126115b94a56a64104b9cbd76e8d: Generated new TSK 0
I20260810 22:34:05.245517 23058 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260810 22:34:05.284597 23074 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260810 22:34:05.284926 23074 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 22:34:05.285528 23045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:05.262659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40886 (local address 127.16.233.126:41179)
0810 22:34:05.262834 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:05.262840 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:05.263689 (+ 849us) server_negotiation.cc:407] Connection header received
0810 22:34:05.264096 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:05.264100 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:05.264170 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:05.264297 (+ 127us) server_negotiation.cc:226] Negotiated authn=JWT
0810 22:34:05.264618 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:05.265483 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:05.266671 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:05.267022 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:05.267133 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:05.267558 (+ 425us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 22:34:05.285035 (+ 17477us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34501/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0810 22:34:05.285376 (+ 341us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40886: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34501/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":74}
W20260810 22:34:05.285601 23045 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40886: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34501/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260810 22:34:05.286968 23073 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:34:05.262483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:41179 (local address 127.0.0.1:40886)
0810 22:34:05.263549 (+ 1066us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:05.263590 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:05.263846 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:05.264349 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:05.264352 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:05.264363 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0810 22:34:05.264488 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:05.264493 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:05.265627 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:05.265631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:05.266532 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:05.266545 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:05.267175 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:05.267179 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:05.267258 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:05.267323 (+ 65us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 22:34:05.285348 (+ 18025us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34501/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0810 22:34:05.285621 (+ 273us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.233.126:41179: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34501/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":923,"thread_start_us":793,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.16.233.126:41179: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34501/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260810 22:34:05.287863 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 22963
2026-08-10T22:34:05Z chronyd exiting
I20260810 22:34:05.330333 17317 test_util.cc:175] -----------------------------------------------
I20260810 22:34:05.330438 17317 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786401215402979-17317-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (363 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260810 22:34:05.331796 17317 mini_oidc.cc:152] Starting JWKS server
I20260810 22:34:05.332657 17317 webserver.cc:460] Webserver started at https://127.0.0.1:38707/ using document root <none> and password file <none>
I20260810 22:34:05.332751 17317 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 22:34:05.333112 17317 webserver.cc:460] Webserver started at http://127.0.0.1:33967/ using document root <none> and password file <none>
2026-08-10T22:34:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:34:05Z Disabled control of system clock
I20260810 22:34:05.345077 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786401215402979-17317-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786401215402979-17317-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786401215402979-17317-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:43973
--webserver_interface=127.16.233.126
--webserver_port=0
--builtin_ntp_servers=127.16.233.84:44859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.126:43973
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38707/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786401215402979-17317-0/testchainca.cert with env {}
W20260810 22:34:05.480623 23080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:34:05.481071 23080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:34:05.481125 23080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:34:05.485504 23080 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:34:05.485582 23080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:34:05.485598 23080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:34:05.485617 23080 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:34:05.485633 23080 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 22:34:05.485649 23080 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 22:34:05.485663 23080 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38707/jwks.json
I20260810 22:34:05.491842 23080 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:44859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786401215402979-17317-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786401215402979-17317-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.126:43973
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:43973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38707/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786401215402979-17317-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786401215402979-17317-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:34:05.493172 23080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:34:05.494256 23080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:34:05.505004 23088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:05.504964 23087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:05.505144 23090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:05.505460 23080 server_base.cc:1034] running on GCE node
I20260810 22:34:05.506168 23080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:34:05.507484 23080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:34:05.508733 23080 hybrid_clock.cc:648] HybridClock initialized: now 1786401245508694 us; error 53 us; skew 500 ppm
I20260810 22:34:05.511171 23080 webserver.cc:460] Webserver started at http://127.16.233.126:46269/ using document root <none> and password file <none>
I20260810 22:34:05.511860 23080 fs_manager.cc:359] Metadata directory not provided
I20260810 22:34:05.511970 23080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:34:05.512231 23080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:34:05.514839 23080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786401215402979-17317-0/minicluster-data/master-0/data/instance:
uuid: "003dc54a64374c728a82f4c9f52750d7"
format_stamp: "Formatted at 2026-08-10 22:34:05 on dist-test-slave-77v9"
server_key: "712de66ff1769ba427916f56f41467f9"
server_key_iv: "57e50ed084e2910d759089a3cbfbe281"
server_key_version: "encryptionkey@0"
I20260810 22:34:05.515594 23080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786401215402979-17317-0/minicluster-data/master-0/wal/instance:
uuid: "003dc54a64374c728a82f4c9f52750d7"
format_stamp: "Formatted at 2026-08-10 22:34:05 on dist-test-slave-77v9"
server_key: "712de66ff1769ba427916f56f41467f9"
server_key_iv: "57e50ed084e2910d759089a3cbfbe281"
server_key_version: "encryptionkey@0"
I20260810 22:34:05.520454 23080 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260810 22:34:05.524039 23096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:05.525720 23080 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:34:05.525882 23080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786401215402979-17317-0/minicluster-data/master-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786401215402979-17317-0/minicluster-data/master-0/wal
uuid: "003dc54a64374c728a82f4c9f52750d7"
format_stamp: "Formatted at 2026-08-10 22:34:05 on dist-test-slave-77v9"
server_key: "712de66ff1769ba427916f56f41467f9"
server_key_iv: "57e50ed084e2910d759089a3cbfbe281"
server_key_version: "encryptionkey@0"
I20260810 22:34:05.526057 23080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786401215402979-17317-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786401215402979-17317-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786401215402979-17317-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:34:05.544379 23080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:34:05.545354 23080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:34:05.545580 23080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:34:05.553643 23080 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.126:43973
I20260810 22:34:05.553668 23152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.126:43973 every 8 connection(s)
I20260810 22:34:05.554792 23080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
I20260810 22:34:05.558329 23153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:34:05.564471 23153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 003dc54a64374c728a82f4c9f52750d7: Bootstrap starting.
I20260810 22:34:05.565059 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 23080
I20260810 22:34:05.565168 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786401215402979-17317-0/minicluster-data/master-0/wal/instance
I20260810 22:34:05.565495 17317 external_mini_cluster.cc:1428] Setting key 5b07cc45db5cb18e0dbb457cde3e4dd3
I20260810 22:34:05.567548 23153 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 003dc54a64374c728a82f4c9f52750d7: Neither blocks nor log segments found. Creating new log.
I20260810 22:34:05.568423 23153 log.cc:824] T 00000000000000000000000000000000 P 003dc54a64374c728a82f4c9f52750d7: Log is configured to *not* fsync() on all Append() calls
I20260810 22:34:05.570686 23153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 003dc54a64374c728a82f4c9f52750d7: No bootstrap required, opened a new log
I20260810 22:34:05.574883 23153 raft_consensus.cc:348] T 00000000000000000000000000000000 P 003dc54a64374c728a82f4c9f52750d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "003dc54a64374c728a82f4c9f52750d7" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 43973 } }
I20260810 22:34:05.575232 23153 raft_consensus.cc:374] T 00000000000000000000000000000000 P 003dc54a64374c728a82f4c9f52750d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:34:05.575335 23153 raft_consensus.cc:729] T 00000000000000000000000000000000 P 003dc54a64374c728a82f4c9f52750d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 003dc54a64374c728a82f4c9f52750d7, State: Initialized, Role: FOLLOWER
I20260810 22:34:05.575847 23153 consensus_queue.cc:260] T 00000000000000000000000000000000 P 003dc54a64374c728a82f4c9f52750d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "003dc54a64374c728a82f4c9f52750d7" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 43973 } }
I20260810 22:34:05.576017 23153 raft_consensus.cc:388] T 00000000000000000000000000000000 P 003dc54a64374c728a82f4c9f52750d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:34:05.576054 23156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:05.566913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50306 (local address 127.16.233.126:43973)
0810 22:34:05.567697 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:05.567720 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:05.567777 (+ 57us) server_negotiation.cc:407] Connection header received
0810 22:34:05.568520 (+ 743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:05.568541 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:05.569119 (+ 578us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:05.569497 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:05.569969 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:05.571087 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:05.572057 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:05.572569 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:05.572779 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:05.573843 (+ 1064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:05.573849 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:05.573865 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:34:05.574332 (+ 467us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:05.574405 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:34:05.574427 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:05.574543 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:05.574699 (+ 156us) server_negotiation.cc:299] Negotiation successful
0810 22:34:05.574999 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":149,"threads_started":1}
I20260810 22:34:05.576097 23153 raft_consensus.cc:482] T 00000000000000000000000000000000 P 003dc54a64374c728a82f4c9f52750d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:34:05.576213 23153 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 003dc54a64374c728a82f4c9f52750d7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:34:05.577567 23153 raft_consensus.cc:504] T 00000000000000000000000000000000 P 003dc54a64374c728a82f4c9f52750d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "003dc54a64374c728a82f4c9f52750d7" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 43973 } }
I20260810 22:34:05.577968 23153 leader_election.cc:302] T 00000000000000000000000000000000 P 003dc54a64374c728a82f4c9f52750d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 003dc54a64374c728a82f4c9f52750d7; no voters:
I20260810 22:34:05.578380 23153 leader_election.cc:288] T 00000000000000000000000000000000 P 003dc54a64374c728a82f4c9f52750d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:34:05.578562 23158 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 003dc54a64374c728a82f4c9f52750d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:34:05.578799 23158 raft_consensus.cc:686] T 00000000000000000000000000000000 P 003dc54a64374c728a82f4c9f52750d7 [term 1 LEADER]: Becoming Leader. State: Replica: 003dc54a64374c728a82f4c9f52750d7, State: Running, Role: LEADER
I20260810 22:34:05.579145 23158 consensus_queue.cc:237] T 00000000000000000000000000000000 P 003dc54a64374c728a82f4c9f52750d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "003dc54a64374c728a82f4c9f52750d7" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 43973 } }
I20260810 22:34:05.579453 17317 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 22:34:05.579816 23153 sys_catalog.cc:564] T 00000000000000000000000000000000 P 003dc54a64374c728a82f4c9f52750d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:34:05.580440 23158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 003dc54a64374c728a82f4c9f52750d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 003dc54a64374c728a82f4c9f52750d7. Latest consensus state: current_term: 1 leader_uuid: "003dc54a64374c728a82f4c9f52750d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "003dc54a64374c728a82f4c9f52750d7" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 43973 } } }
I20260810 22:34:05.580880 23158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 003dc54a64374c728a82f4c9f52750d7 [sys.catalog]: This master's current role is: LEADER
I20260810 22:34:05.581313 23159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 003dc54a64374c728a82f4c9f52750d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "003dc54a64374c728a82f4c9f52750d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "003dc54a64374c728a82f4c9f52750d7" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 43973 } } }
I20260810 22:34:05.581687 23159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 003dc54a64374c728a82f4c9f52750d7 [sys.catalog]: This master's current role is: LEADER
I20260810 22:34:05.581763 23167 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:34:05.585034 23167 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:34:05.591747 23167 catalog_manager.cc:1347] Generated new cluster ID: db5390d9fbe54b9681278bf6a47b680d
I20260810 22:34:05.591847 23167 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:34:05.603832 23167 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:34:05.605252 23167 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:34:05.615908 23167 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 003dc54a64374c728a82f4c9f52750d7: Generated new TSK 0
I20260810 22:34:05.616798 23167 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260810 22:34:05.649013 23185 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260810 22:34:05.649138 23185 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 22:34:05.649772 23156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:05.632427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50314 (local address 127.16.233.126:43973)
0810 22:34:05.632599 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:05.632604 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:05.632844 (+ 240us) server_negotiation.cc:407] Connection header received
0810 22:34:05.633074 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:05.633077 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:05.633143 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:05.633274 (+ 131us) server_negotiation.cc:226] Negotiated authn=JWT
0810 22:34:05.633551 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:05.634463 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:05.635534 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:05.635894 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:05.636007 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:05.636382 (+ 375us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 22:34:05.649283 (+ 12901us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38707/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0810 22:34:05.649618 (+ 335us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50314: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38707/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":70}
W20260810 22:34:05.649856 23156 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50314: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38707/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260810 22:34:05.650149 23184 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:34:05.632306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:43973 (local address 127.0.0.1:50314)
0810 22:34:05.632680 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:05.632703 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:05.632913 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:05.633295 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:05.633297 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:05.633306 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0810 22:34:05.633425 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:05.633431 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:05.634623 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:05.634626 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:05.635394 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:05.635405 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:05.636047 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:05.636052 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:05.636128 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:05.636159 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 22:34:05.649569 (+ 13410us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38707/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0810 22:34:05.649898 (+ 329us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.233.126:43973: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38707/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":128,"threads_started":1}
I20260810 22:34:05.650993 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 23080
2026-08-10T22:34:05Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (366 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260810 22:34:05.698405 17317 mini_oidc.cc:152] Starting JWKS server
I20260810 22:34:05.699425 17317 webserver.cc:460] Webserver started at https://127.0.0.1:40579/ using document root <none> and password file <none>
I20260810 22:34:05.699576 17317 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 22:34:05.699928 17317 webserver.cc:460] Webserver started at http://127.0.0.1:46361/ using document root <none> and password file <none>
2026-08-10T22:34:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:34:05Z Disabled control of system clock
I20260810 22:34:05.711774 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786401215402979-17317-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786401215402979-17317-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786401215402979-17317-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:41211
--webserver_interface=127.16.233.126
--webserver_port=0
--builtin_ntp_servers=127.16.233.84:36489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.126:41211
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40579/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260810 22:34:05.833534 23191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:34:05.833870 23191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:34:05.833914 23191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:34:05.838372 23191 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:34:05.838501 23191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:34:05.838531 23191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:34:05.838564 23191 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:34:05.838624 23191 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 22:34:05.838665 23191 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 22:34:05.838699 23191 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40579/jwks.json
I20260810 22:34:05.843999 23191 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:36489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786401215402979-17317-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786401215402979-17317-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.126:41211
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:41211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40579/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786401215402979-17317-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:34:05.845417 23191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:34:05.847007 23191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:34:05.856943 23202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:05.856959 23198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:05.856947 23199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:05.857276 23191 server_base.cc:1034] running on GCE node
I20260810 22:34:05.858151 23191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:34:05.859599 23191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:34:05.860965 23191 hybrid_clock.cc:648] HybridClock initialized: now 1786401245860945 us; error 47 us; skew 500 ppm
I20260810 22:34:05.863274 23191 webserver.cc:460] Webserver started at http://127.16.233.126:38185/ using document root <none> and password file <none>
I20260810 22:34:05.863945 23191 fs_manager.cc:359] Metadata directory not provided
I20260810 22:34:05.864060 23191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:34:05.864329 23191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:34:05.866724 23191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786401215402979-17317-0/minicluster-data/master-0/data/instance:
uuid: "bc380b9df3ba420e8a5c2a4cceff201a"
format_stamp: "Formatted at 2026-08-10 22:34:05 on dist-test-slave-77v9"
server_key: "2fe3be8d1acaefe357697f027eecfd61"
server_key_iv: "7de69156893902eef3afab7a73f8ddaf"
server_key_version: "encryptionkey@0"
I20260810 22:34:05.867427 23191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786401215402979-17317-0/minicluster-data/master-0/wal/instance:
uuid: "bc380b9df3ba420e8a5c2a4cceff201a"
format_stamp: "Formatted at 2026-08-10 22:34:05 on dist-test-slave-77v9"
server_key: "2fe3be8d1acaefe357697f027eecfd61"
server_key_iv: "7de69156893902eef3afab7a73f8ddaf"
server_key_version: "encryptionkey@0"
I20260810 22:34:05.871759 23191 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 22:34:05.874815 23211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:05.876260 23191 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 22:34:05.876441 23191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786401215402979-17317-0/minicluster-data/master-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786401215402979-17317-0/minicluster-data/master-0/wal
uuid: "bc380b9df3ba420e8a5c2a4cceff201a"
format_stamp: "Formatted at 2026-08-10 22:34:05 on dist-test-slave-77v9"
server_key: "2fe3be8d1acaefe357697f027eecfd61"
server_key_iv: "7de69156893902eef3afab7a73f8ddaf"
server_key_version: "encryptionkey@0"
I20260810 22:34:05.876587 23191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786401215402979-17317-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786401215402979-17317-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786401215402979-17317-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:34:05.898773 23191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:34:05.899662 23191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:34:05.899909 23191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:34:05.907953 23191 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.126:41211
I20260810 22:34:05.907979 23266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.126:41211 every 8 connection(s)
I20260810 22:34:05.909101 23191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
I20260810 22:34:05.909866 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 23191
I20260810 22:34:05.909972 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786401215402979-17317-0/minicluster-data/master-0/wal/instance
I20260810 22:34:05.910274 17317 external_mini_cluster.cc:1428] Setting key 05c994a730e0c5c97d43552854c6d74b
I20260810 22:34:05.913870 23267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:34:05.920090 23267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc380b9df3ba420e8a5c2a4cceff201a: Bootstrap starting.
I20260810 22:34:05.920197 23270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:05.911665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33852 (local address 127.16.233.126:41211)
0810 22:34:05.912335 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:05.912355 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:05.912396 (+ 41us) server_negotiation.cc:407] Connection header received
0810 22:34:05.913200 (+ 804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:05.913221 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:05.913643 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:05.914098 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:05.914568 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:05.915580 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:05.916543 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:05.917150 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:05.917339 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:05.918328 (+ 989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:05.918333 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:05.918346 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:34:05.918761 (+ 415us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:05.918834 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:34:05.918856 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:05.918978 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:05.919122 (+ 144us) server_negotiation.cc:299] Negotiation successful
0810 22:34:05.919410 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":145,"threads_started":1}
I20260810 22:34:05.923169 23267 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc380b9df3ba420e8a5c2a4cceff201a: Neither blocks nor log segments found. Creating new log.
I20260810 22:34:05.923532 17317 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 22:34:05.924095 23267 log.cc:824] T 00000000000000000000000000000000 P bc380b9df3ba420e8a5c2a4cceff201a: Log is configured to *not* fsync() on all Append() calls
I20260810 22:34:05.926359 23267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc380b9df3ba420e8a5c2a4cceff201a: No bootstrap required, opened a new log
I20260810 22:34:05.930256 23267 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc380b9df3ba420e8a5c2a4cceff201a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc380b9df3ba420e8a5c2a4cceff201a" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 41211 } }
I20260810 22:34:05.930572 23267 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc380b9df3ba420e8a5c2a4cceff201a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:34:05.930660 23267 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc380b9df3ba420e8a5c2a4cceff201a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc380b9df3ba420e8a5c2a4cceff201a, State: Initialized, Role: FOLLOWER
I20260810 22:34:05.931152 23267 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc380b9df3ba420e8a5c2a4cceff201a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc380b9df3ba420e8a5c2a4cceff201a" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 41211 } }
I20260810 22:34:05.931288 23267 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc380b9df3ba420e8a5c2a4cceff201a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:34:05.931360 23267 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc380b9df3ba420e8a5c2a4cceff201a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:34:05.931501 23267 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc380b9df3ba420e8a5c2a4cceff201a [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:34:05.932554 23267 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc380b9df3ba420e8a5c2a4cceff201a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc380b9df3ba420e8a5c2a4cceff201a" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 41211 } }
I20260810 22:34:05.932999 23267 leader_election.cc:302] T 00000000000000000000000000000000 P bc380b9df3ba420e8a5c2a4cceff201a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc380b9df3ba420e8a5c2a4cceff201a; no voters:
I20260810 22:34:05.933384 23267 leader_election.cc:288] T 00000000000000000000000000000000 P bc380b9df3ba420e8a5c2a4cceff201a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:34:05.933617 23272 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc380b9df3ba420e8a5c2a4cceff201a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:34:05.933890 23272 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc380b9df3ba420e8a5c2a4cceff201a [term 1 LEADER]: Becoming Leader. State: Replica: bc380b9df3ba420e8a5c2a4cceff201a, State: Running, Role: LEADER
I20260810 22:34:05.934243 23272 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc380b9df3ba420e8a5c2a4cceff201a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc380b9df3ba420e8a5c2a4cceff201a" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 41211 } }
I20260810 22:34:05.934901 23267 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc380b9df3ba420e8a5c2a4cceff201a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:34:05.936106 23274 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc380b9df3ba420e8a5c2a4cceff201a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc380b9df3ba420e8a5c2a4cceff201a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc380b9df3ba420e8a5c2a4cceff201a" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 41211 } } }
I20260810 22:34:05.936254 23274 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc380b9df3ba420e8a5c2a4cceff201a [sys.catalog]: This master's current role is: LEADER
I20260810 22:34:05.936677 23282 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:34:05.937785 23275 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc380b9df3ba420e8a5c2a4cceff201a [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc380b9df3ba420e8a5c2a4cceff201a. Latest consensus state: current_term: 1 leader_uuid: "bc380b9df3ba420e8a5c2a4cceff201a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc380b9df3ba420e8a5c2a4cceff201a" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 41211 } } }
I20260810 22:34:05.937907 23275 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc380b9df3ba420e8a5c2a4cceff201a [sys.catalog]: This master's current role is: LEADER
I20260810 22:34:05.939822 23282 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:34:05.946813 23282 catalog_manager.cc:1347] Generated new cluster ID: 3431be8f18224781a1832646e2d3b97f
I20260810 22:34:05.946913 23282 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:34:05.962764 23282 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:34:05.963846 23282 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:34:05.977228 23282 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc380b9df3ba420e8a5c2a4cceff201a: Generated new TSK 0
I20260810 22:34:05.978276 23282 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:34:06.002130 23270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:05.994823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33856 (local address 127.16.233.126:41211)
0810 22:34:05.995005 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:05.995010 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:05.995204 (+ 194us) server_negotiation.cc:407] Connection header received
0810 22:34:05.995397 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:05.995401 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:05.995470 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:05.995660 (+ 190us) server_negotiation.cc:226] Negotiated authn=JWT
0810 22:34:05.995915 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:05.996760 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:05.997920 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:05.998285 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:05.998407 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:05.998827 (+ 420us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 22:34:06.001667 (+ 2840us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40579/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0810 22:34:06.001966 (+ 299us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33856: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40579/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":74}
W20260810 22:34:06.002219 23270 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33856: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40579/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260810 22:34:06.003980 23297 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:34:05.994652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:41211 (local address 127.0.0.1:33856)
0810 22:34:05.995106 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:05.995119 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:05.995232 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:05.995674 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:05.995676 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:05.995684 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0810 22:34:05.995793 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:05.995798 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:05.996991 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:05.996995 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:05.997774 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:05.997784 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:05.998488 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:05.998491 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:05.998572 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:05.998602 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 22:34:06.001896 (+ 3294us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40579/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0810 22:34:06.003794 (+ 1898us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.233.126:41211: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40579/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":331,"mutex_wait_us":1635,"thread_start_us":144,"threads_started":1}
I20260810 22:34:06.004839 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 23191
E20260810 22:34:06.014632 23298 webserver.cc:533] Webserver: sslize failed:
E20260810 22:34:06.014730 23298 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-10T22:34:06Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (350 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-10T22:34:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:34:06Z Disabled control of system clock
I20260810 22:34:06.061209 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786401215402979-17317-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786401215402979-17317-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786401215402979-17317-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:37561
--webserver_interface=127.16.233.126
--webserver_port=0
--builtin_ntp_servers=127.16.233.84:35061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.126:37561
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786401215402979-17317-0/testchainca.cert with env {}
W20260810 22:34:06.184581 23302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:34:06.184934 23302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:34:06.184983 23302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:34:06.189399 23302 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:34:06.189484 23302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:34:06.189499 23302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:34:06.189517 23302 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:34:06.189533 23302 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 22:34:06.189550 23302 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 22:34:06.189564 23302 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260810 22:34:06.195573 23302 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:35061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786401215402979-17317-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786401215402979-17317-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.126:37561
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:37561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786401215402979-17317-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786401215402979-17317-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:34:06.196892 23302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:34:06.198081 23302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:34:06.208004 23307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:06.208020 23308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:06.208117 23302 server_base.cc:1034] running on GCE node
W20260810 22:34:06.208045 23310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:06.208999 23302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:34:06.210290 23302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:34:06.211483 23302 hybrid_clock.cc:648] HybridClock initialized: now 1786401246211461 us; error 37 us; skew 500 ppm
I20260810 22:34:06.213828 23302 webserver.cc:460] Webserver started at http://127.16.233.126:35345/ using document root <none> and password file <none>
I20260810 22:34:06.214533 23302 fs_manager.cc:359] Metadata directory not provided
I20260810 22:34:06.214614 23302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:34:06.214877 23302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:34:06.217509 23302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786401215402979-17317-0/minicluster-data/master-0/data/instance:
uuid: "5609a77cff5147c7ad92f9353b2215b2"
format_stamp: "Formatted at 2026-08-10 22:34:06 on dist-test-slave-77v9"
server_key: "64f9420b7fcbc8aae621d287aeedead2"
server_key_iv: "2ec0bec55221699ca6722c86a3645e8b"
server_key_version: "encryptionkey@0"
I20260810 22:34:06.218281 23302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786401215402979-17317-0/minicluster-data/master-0/wal/instance:
uuid: "5609a77cff5147c7ad92f9353b2215b2"
format_stamp: "Formatted at 2026-08-10 22:34:06 on dist-test-slave-77v9"
server_key: "64f9420b7fcbc8aae621d287aeedead2"
server_key_iv: "2ec0bec55221699ca6722c86a3645e8b"
server_key_version: "encryptionkey@0"
I20260810 22:34:06.222921 23302 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 22:34:06.226110 23316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:06.227852 23302 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 22:34:06.228080 23302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786401215402979-17317-0/minicluster-data/master-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786401215402979-17317-0/minicluster-data/master-0/wal
uuid: "5609a77cff5147c7ad92f9353b2215b2"
format_stamp: "Formatted at 2026-08-10 22:34:06 on dist-test-slave-77v9"
server_key: "64f9420b7fcbc8aae621d287aeedead2"
server_key_iv: "2ec0bec55221699ca6722c86a3645e8b"
server_key_version: "encryptionkey@0"
I20260810 22:34:06.228245 23302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786401215402979-17317-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786401215402979-17317-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786401215402979-17317-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:34:06.261540 23302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:34:06.262425 23302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:34:06.262679 23302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:34:06.270756 23302 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.126:37561
I20260810 22:34:06.270778 23375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.126:37561 every 8 connection(s)
I20260810 22:34:06.271893 23302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
I20260810 22:34:06.275347 23376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:34:06.280372 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 23302
I20260810 22:34:06.280526 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786401215402979-17317-0/minicluster-data/master-0/wal/instance
I20260810 22:34:06.280874 17317 external_mini_cluster.cc:1428] Setting key 4ed3682155e1e280cc0bf8ad84c7c0f8
I20260810 22:34:06.282558 23376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5609a77cff5147c7ad92f9353b2215b2: Bootstrap starting.
I20260810 22:34:06.285382 23376 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5609a77cff5147c7ad92f9353b2215b2: Neither blocks nor log segments found. Creating new log.
I20260810 22:34:06.286270 23376 log.cc:824] T 00000000000000000000000000000000 P 5609a77cff5147c7ad92f9353b2215b2: Log is configured to *not* fsync() on all Append() calls
I20260810 22:34:06.288571 23376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5609a77cff5147c7ad92f9353b2215b2: No bootstrap required, opened a new log
I20260810 22:34:06.291271 23379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:06.282534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59800 (local address 127.16.233.126:37561)
0810 22:34:06.283282 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:06.283317 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:06.283356 (+ 39us) server_negotiation.cc:407] Connection header received
0810 22:34:06.284119 (+ 763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:06.284139 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:06.284587 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:06.285005 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:06.285533 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:06.286642 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:06.287590 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:06.288090 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:06.288328 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:06.289360 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:06.289366 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:06.289380 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:34:06.289829 (+ 449us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:06.289888 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:34:06.289909 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:06.290027 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:06.290157 (+ 130us) server_negotiation.cc:299] Negotiation successful
0810 22:34:06.290466 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":166,"threads_started":1}
I20260810 22:34:06.293340 23376 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5609a77cff5147c7ad92f9353b2215b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5609a77cff5147c7ad92f9353b2215b2" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 37561 } }
I20260810 22:34:06.294070 23376 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5609a77cff5147c7ad92f9353b2215b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:34:06.294157 23376 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5609a77cff5147c7ad92f9353b2215b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5609a77cff5147c7ad92f9353b2215b2, State: Initialized, Role: FOLLOWER
I20260810 22:34:06.294687 23376 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5609a77cff5147c7ad92f9353b2215b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5609a77cff5147c7ad92f9353b2215b2" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 37561 } }
I20260810 22:34:06.294775 17317 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 22:34:06.294875 23376 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5609a77cff5147c7ad92f9353b2215b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:34:06.294948 23376 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5609a77cff5147c7ad92f9353b2215b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:34:06.295053 23376 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5609a77cff5147c7ad92f9353b2215b2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:34:06.296257 23376 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5609a77cff5147c7ad92f9353b2215b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5609a77cff5147c7ad92f9353b2215b2" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 37561 } }
I20260810 22:34:06.296676 23376 leader_election.cc:302] T 00000000000000000000000000000000 P 5609a77cff5147c7ad92f9353b2215b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5609a77cff5147c7ad92f9353b2215b2; no voters:
I20260810 22:34:06.297104 23376 leader_election.cc:288] T 00000000000000000000000000000000 P 5609a77cff5147c7ad92f9353b2215b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:34:06.297276 23381 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5609a77cff5147c7ad92f9353b2215b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:34:06.297556 23381 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5609a77cff5147c7ad92f9353b2215b2 [term 1 LEADER]: Becoming Leader. State: Replica: 5609a77cff5147c7ad92f9353b2215b2, State: Running, Role: LEADER
I20260810 22:34:06.297892 23381 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5609a77cff5147c7ad92f9353b2215b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5609a77cff5147c7ad92f9353b2215b2" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 37561 } }
I20260810 22:34:06.298445 23376 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5609a77cff5147c7ad92f9353b2215b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:34:06.299667 23382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5609a77cff5147c7ad92f9353b2215b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5609a77cff5147c7ad92f9353b2215b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5609a77cff5147c7ad92f9353b2215b2" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 37561 } } }
I20260810 22:34:06.299821 23382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5609a77cff5147c7ad92f9353b2215b2 [sys.catalog]: This master's current role is: LEADER
I20260810 22:34:06.300199 23383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5609a77cff5147c7ad92f9353b2215b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5609a77cff5147c7ad92f9353b2215b2. Latest consensus state: current_term: 1 leader_uuid: "5609a77cff5147c7ad92f9353b2215b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5609a77cff5147c7ad92f9353b2215b2" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 37561 } } }
I20260810 22:34:06.300297 23383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5609a77cff5147c7ad92f9353b2215b2 [sys.catalog]: This master's current role is: LEADER
W20260810 22:34:06.303552 23398 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5609a77cff5147c7ad92f9353b2215b2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 22:34:06.303637 23398 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 22:34:06.303722 23392 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:34:06.304725 23392 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:34:06.310638 23392 catalog_manager.cc:1347] Generated new cluster ID: 622bbff83a074702a71029593861a99f
I20260810 22:34:06.310738 23392 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:34:06.320649 23392 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:34:06.321909 23392 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:34:06.346719 23392 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5609a77cff5147c7ad92f9353b2215b2: Generated new TSK 0
I20260810 22:34:06.347735 23392 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:34:06.465867 23379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:06.367139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59816 (local address 127.16.233.126:37561)
0810 22:34:06.367312 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:06.367318 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:06.367485 (+ 167us) server_negotiation.cc:407] Connection header received
0810 22:34:06.367668 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:06.367672 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:06.367762 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:06.367894 (+ 132us) server_negotiation.cc:226] Negotiated authn=JWT
0810 22:34:06.368198 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:06.369087 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:06.370051 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:06.370440 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:06.370549 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:06.370907 (+ 358us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 22:34:06.465304 (+ 94397us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0810 22:34:06.465720 (+ 416us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59816: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":61}
W20260810 22:34:06.465970 23379 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59816: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260810 22:34:06.466110 23407 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:34:06.366970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:37561 (local address 127.0.0.1:59816)
0810 22:34:06.367379 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:06.367393 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:06.367515 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:06.367931 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:06.367934 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:06.367942 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0810 22:34:06.368065 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:06.368071 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:06.369256 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:06.369259 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:06.369923 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:06.369931 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:06.370582 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:06.370585 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:06.370674 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:06.370702 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 22:34:06.465620 (+ 94918us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0810 22:34:06.465830 (+ 210us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.233.126:37561: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":139,"threads_started":1}
I20260810 22:34:06.467008 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 23302
2026-08-10T22:34:06Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (443 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (144 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (147 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:34:06 dist-test-slave-77v9 krb5kdc[23421](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:34:06 dist-test-slave-77v9 krb5kdc[23421](info): set up 2 sockets
Aug 10 22:34:06 dist-test-slave-77v9 krb5kdc[23421](info): commencing operation
krb5kdc: starting...
W20260810 22:34:07.970798 17317 mini_kdc.cc:121] Time spent starting KDC: real 1.186s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Aug 10 22:34:08 dist-test-slave-77v9 krb5kdc[23421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:34:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:34:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.233.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab.
Entry for principal kudu/127.16.233.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab.
I20260810 22:34:08.130499 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:45327
--webserver_interface=127.16.233.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab
--principal=kudu/127.16.233.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:40843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.126:45327
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:34:08.256129 23434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:34:08.256551 23434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:34:08.256641 23434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:34:08.261142 23434 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:34:08.261281 23434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:34:08.261318 23434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:34:08.261363 23434 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:34:08.261417 23434 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:34:08.267396 23434 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:40843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.126:45327
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab
--principal=kudu/127.16.233.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:45327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23434
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:34:08.268695 23434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:34:08.269903 23434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:34:08.279606 23440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:08.279604 23439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:08.279878 23442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:08.280057 23434 server_base.cc:1034] running on GCE node
I20260810 22:34:08.280954 23434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:34:08.282294 23434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:34:08.283521 23434 hybrid_clock.cc:648] HybridClock initialized: now 1786401248283513 us; error 36 us; skew 500 ppm
Aug 10 22:34:08 dist-test-slave-77v9 krb5kdc[23421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401248, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:34:08.291735 23434 init.cc:377] Logged in from keytab as kudu/127.16.233.126@KRBTEST.COM (short username kudu)
I20260810 22:34:08.293179 23434 webserver.cc:460] Webserver started at http://127.16.233.126:44989/ using document root <none> and password file <none>
I20260810 22:34:08.293852 23434 fs_manager.cc:359] Metadata directory not provided
I20260810 22:34:08.293984 23434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:34:08.294261 23434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:34:08.296782 23434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/minicluster-data/master-0/data/instance:
uuid: "d108df65139b43f69edb67f6161e246f"
format_stamp: "Formatted at 2026-08-10 22:34:08 on dist-test-slave-77v9"
server_key: "791604e2434edae8150eeffca644677a"
server_key_iv: "4a06daee405190cea244ff11def11837"
server_key_version: "encryptionkey@0"
I20260810 22:34:08.297683 23434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/minicluster-data/master-0/wal/instance:
uuid: "d108df65139b43f69edb67f6161e246f"
format_stamp: "Formatted at 2026-08-10 22:34:08 on dist-test-slave-77v9"
server_key: "791604e2434edae8150eeffca644677a"
server_key_iv: "4a06daee405190cea244ff11def11837"
server_key_version: "encryptionkey@0"
I20260810 22:34:08.302405 23434 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260810 22:34:08.305629 23450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:08.307435 23434 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 22:34:08.307653 23434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/minicluster-data/master-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/minicluster-data/master-0/wal
uuid: "d108df65139b43f69edb67f6161e246f"
format_stamp: "Formatted at 2026-08-10 22:34:08 on dist-test-slave-77v9"
server_key: "791604e2434edae8150eeffca644677a"
server_key_iv: "4a06daee405190cea244ff11def11837"
server_key_version: "encryptionkey@0"
I20260810 22:34:08.307819 23434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:34:08.333647 23434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:34:08.337932 23434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:34:08.338196 23434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:34:08.347044 23434 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.126:45327
I20260810 22:34:08.347078 23513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.126:45327 every 8 connection(s)
I20260810 22:34:08.348260 23434 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
I20260810 22:34:08.350629 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 23434
I20260810 22:34:08.350775 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/minicluster-data/master-0/wal/instance
I20260810 22:34:08.351137 17317 external_mini_cluster.cc:1428] Setting key 533c2ec86964f0c23f24c5d68c6e4d50
I20260810 22:34:08.352175 23514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:34:08.359315 23514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d108df65139b43f69edb67f6161e246f: Bootstrap starting.
Aug 10 22:34:08 dist-test-slave-77v9 krb5kdc[23421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:34:08.362433 23514 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d108df65139b43f69edb67f6161e246f: Neither blocks nor log segments found. Creating new log.
I20260810 22:34:08.363422 23514 log.cc:824] T 00000000000000000000000000000000 P d108df65139b43f69edb67f6161e246f: Log is configured to *not* fsync() on all Append() calls
I20260810 22:34:08.366437 23514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d108df65139b43f69edb67f6161e246f: No bootstrap required, opened a new log
I20260810 22:34:08.368641 23517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:08.353094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51646 (local address 127.16.233.126:45327)
0810 22:34:08.353668 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:08.353686 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:08.353730 (+ 44us) server_negotiation.cc:407] Connection header received
0810 22:34:08.354550 (+ 820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:08.354572 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:08.355007 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:08.355479 (+ 472us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:08.356461 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:08.357689 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:08.358756 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:08.359349 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:08.359569 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:08.362863 (+ 3294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:08.362871 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:08.362890 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:08.362935 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:08.365146 (+ 2211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:08.365782 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:08.365787 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:08.365792 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:08.365998 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:08.366335 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:08.366338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:08.366340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:08.366931 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:08.367089 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:08.367433 (+ 344us) server_negotiation.cc:299] Negotiation successful
0810 22:34:08.367763 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":194,"threads_started":1}
I20260810 22:34:08.370646 23514 raft_consensus.cc:348] T 00000000000000000000000000000000 P d108df65139b43f69edb67f6161e246f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d108df65139b43f69edb67f6161e246f" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 45327 } }
I20260810 22:34:08.370961 23514 raft_consensus.cc:374] T 00000000000000000000000000000000 P d108df65139b43f69edb67f6161e246f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:34:08.371094 23514 raft_consensus.cc:729] T 00000000000000000000000000000000 P d108df65139b43f69edb67f6161e246f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d108df65139b43f69edb67f6161e246f, State: Initialized, Role: FOLLOWER
I20260810 22:34:08.371675 23514 consensus_queue.cc:260] T 00000000000000000000000000000000 P d108df65139b43f69edb67f6161e246f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d108df65139b43f69edb67f6161e246f" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 45327 } }
I20260810 22:34:08.371816 23514 raft_consensus.cc:388] T 00000000000000000000000000000000 P d108df65139b43f69edb67f6161e246f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:34:08.371868 23514 raft_consensus.cc:482] T 00000000000000000000000000000000 P d108df65139b43f69edb67f6161e246f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:34:08.372026 23514 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d108df65139b43f69edb67f6161e246f [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:34:08.373267 23514 raft_consensus.cc:504] T 00000000000000000000000000000000 P d108df65139b43f69edb67f6161e246f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d108df65139b43f69edb67f6161e246f" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 45327 } }
I20260810 22:34:08.373670 23514 leader_election.cc:302] T 00000000000000000000000000000000 P d108df65139b43f69edb67f6161e246f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d108df65139b43f69edb67f6161e246f; no voters:
I20260810 22:34:08.374061 23514 leader_election.cc:288] T 00000000000000000000000000000000 P d108df65139b43f69edb67f6161e246f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:34:08.374416 23519 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d108df65139b43f69edb67f6161e246f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:34:08.374711 23519 raft_consensus.cc:686] T 00000000000000000000000000000000 P d108df65139b43f69edb67f6161e246f [term 1 LEADER]: Becoming Leader. State: Replica: d108df65139b43f69edb67f6161e246f, State: Running, Role: LEADER
I20260810 22:34:08.375087 23519 consensus_queue.cc:237] T 00000000000000000000000000000000 P d108df65139b43f69edb67f6161e246f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d108df65139b43f69edb67f6161e246f" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 45327 } }
I20260810 22:34:08.375439 23514 sys_catalog.cc:564] T 00000000000000000000000000000000 P d108df65139b43f69edb67f6161e246f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:34:08.377084 23521 sys_catalog.cc:455] T 00000000000000000000000000000000 P d108df65139b43f69edb67f6161e246f [sys.catalog]: SysCatalogTable state changed. Reason: New leader d108df65139b43f69edb67f6161e246f. Latest consensus state: current_term: 1 leader_uuid: "d108df65139b43f69edb67f6161e246f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d108df65139b43f69edb67f6161e246f" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 45327 } } }
I20260810 22:34:08.377339 23521 sys_catalog.cc:458] T 00000000000000000000000000000000 P d108df65139b43f69edb67f6161e246f [sys.catalog]: This master's current role is: LEADER
I20260810 22:34:08.377437 23520 sys_catalog.cc:455] T 00000000000000000000000000000000 P d108df65139b43f69edb67f6161e246f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d108df65139b43f69edb67f6161e246f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d108df65139b43f69edb67f6161e246f" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 45327 } } }
I20260810 22:34:08.377571 23520 sys_catalog.cc:458] T 00000000000000000000000000000000 P d108df65139b43f69edb67f6161e246f [sys.catalog]: This master's current role is: LEADER
I20260810 22:34:08.377838 23532 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:34:08.381147 23532 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:34:08.388096 23532 catalog_manager.cc:1347] Generated new cluster ID: 033ae4ed677a442e9c5daadde8b98d2f
I20260810 22:34:08.388218 23532 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:34:08.395607 23532 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:34:08.397272 23532 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:34:08.413609 23532 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d108df65139b43f69edb67f6161e246f: Generated new TSK 0
I20260810 22:34:08.414569 23532 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.233.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab.
Entry for principal kudu/127.16.233.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab.
I20260810 22:34:08.453460 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.65:0
--local_ip_for_outbound_sockets=127.16.233.65
--webserver_interface=127.16.233.65
--webserver_port=0
--tserver_master_addrs=127.16.233.126:45327
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab
--principal=kudu/127.16.233.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:40843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:34:08.584924 23542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:34:08.585312 23542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:34:08.585397 23542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:34:08.589850 23542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:34:08.589979 23542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:34:08.590085 23542 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.65
I20260810 22:34:08.596026 23542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:40843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab
--principal=kudu/127.16.233.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.233.65
--webserver_port=0
--tserver_master_addrs=127.16.233.126:45327
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.65
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:34:08.597458 23542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:34:08.598613 23542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:34:08.609885 23548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:08.609825 23549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:08.609825 23551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:08.610200 23542 server_base.cc:1034] running on GCE node
I20260810 22:34:08.610733 23542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:34:08.611513 23542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:34:08.612715 23542 hybrid_clock.cc:648] HybridClock initialized: now 1786401248612670 us; error 63 us; skew 500 ppm
Aug 10 22:34:08 dist-test-slave-77v9 krb5kdc[23421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401248, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:34:08.616370 23542 init.cc:377] Logged in from keytab as kudu/127.16.233.65@KRBTEST.COM (short username kudu)
I20260810 22:34:08.617635 23542 webserver.cc:460] Webserver started at http://127.16.233.65:35725/ using document root <none> and password file <none>
I20260810 22:34:08.618321 23542 fs_manager.cc:359] Metadata directory not provided
I20260810 22:34:08.618430 23542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:34:08.618688 23542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:34:08.621165 23542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/minicluster-data/ts-0/data/instance:
uuid: "54fb046439004bebb03fae83cb609a47"
format_stamp: "Formatted at 2026-08-10 22:34:08 on dist-test-slave-77v9"
server_key: "9df09cf1e2d4e7726101b2632ba92f21"
server_key_iv: "a34b6a4a2cde2cdecd82a622e4257b9d"
server_key_version: "encryptionkey@0"
I20260810 22:34:08.621858 23542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/minicluster-data/ts-0/wal/instance:
uuid: "54fb046439004bebb03fae83cb609a47"
format_stamp: "Formatted at 2026-08-10 22:34:08 on dist-test-slave-77v9"
server_key: "9df09cf1e2d4e7726101b2632ba92f21"
server_key_iv: "a34b6a4a2cde2cdecd82a622e4257b9d"
server_key_version: "encryptionkey@0"
I20260810 22:34:08.626264 23542 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260810 22:34:08.629555 23559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:08.631191 23542 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 22:34:08.631410 23542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/minicluster-data/ts-0/wal
uuid: "54fb046439004bebb03fae83cb609a47"
format_stamp: "Formatted at 2026-08-10 22:34:08 on dist-test-slave-77v9"
server_key: "9df09cf1e2d4e7726101b2632ba92f21"
server_key_iv: "a34b6a4a2cde2cdecd82a622e4257b9d"
server_key_version: "encryptionkey@0"
I20260810 22:34:08.631568 23542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:34:08.650194 23542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:34:08.654206 23542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:34:08.654480 23542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:34:08.655206 23542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:34:08.656226 23542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:34:08.656306 23542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:08.656383 23542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:34:08.656435 23542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:08.667665 23542 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.65:34543
I20260810 22:34:08.667745 23685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.65:34543 every 8 connection(s)
I20260810 22:34:08.668768 23542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
I20260810 22:34:08.673342 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 23542
I20260810 22:34:08.673537 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/minicluster-data/ts-0/wal/instance
I20260810 22:34:08.673879 17317 external_mini_cluster.cc:1428] Setting key b7dab6dbc8fecd584b2b98490183050b
Aug 10 22:34:08 dist-test-slave-77v9 krb5kdc[23421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401248, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.65@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.233.66@KRBTEST.COM; defaulting to no policy
I20260810 22:34:08.688406 23517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:08.671077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:50565 (local address 127.16.233.126:45327)
0810 22:34:08.671252 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:08.671256 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:08.672253 (+ 997us) server_negotiation.cc:407] Connection header received
0810 22:34:08.674547 (+ 2294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:08.674551 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:08.674620 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:08.674725 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:08.676653 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:08.677516 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:08.679223 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:08.679501 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:08.679626 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:08.683472 (+ 3846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:08.683478 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:08.683485 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:08.683531 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:08.685777 (+ 2246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:08.686507 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:08.686512 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:08.686514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:08.686622 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:08.687097 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:08.687103 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:08.687108 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:08.687337 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:08.687447 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:08.688115 (+ 668us) server_negotiation.cc:299] Negotiation successful
0810 22:34:08.688265 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260810 22:34:08.689549 23688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:08.671454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:45327 (local address 127.16.233.65:50565)
0810 22:34:08.672055 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:08.672109 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:08.674222 (+ 2113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:08.674937 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:08.674945 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:08.675475 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:08.676426 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:08.676448 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:08.677681 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:08.677684 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:08.679030 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:08.679049 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:08.679692 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:08.679696 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:08.679996 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:08.680850 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:08.680880 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:08.683181 (+ 2301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:34:08.685952 (+ 2771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:08.685958 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:08.685975 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:08.686323 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:08.686784 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:08.686787 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:08.686789 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:08.686986 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:08.687476 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:34:08.687482 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:34:08.687827 (+ 345us) client_negotiation.cc:769] Sending connection context
0810 22:34:08.688054 (+ 227us) client_negotiation.cc:240] Negotiation successful
0810 22:34:08.688429 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":147,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.66@KRBTEST.COM" created.
I20260810 22:34:08.691278 23686 heartbeater.cc:347] Connected to a master server at 127.16.233.126:45327
I20260810 22:34:08.691582 23686 heartbeater.cc:464] Registering TS with master...
I20260810 22:34:08.692173 23686 heartbeater.cc:511] Master 127.16.233.126:45327 requested a full tablet report, sending...
I20260810 22:34:08.694360 23469 ts_manager.cc:194] Registered new tserver with Master: 54fb046439004bebb03fae83cb609a47 (127.16.233.65:34543)
I20260810 22:34:08.695799 23469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.65@KRBTEST.COM'} at 127.16.233.65:50565
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab.
Entry for principal kudu/127.16.233.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab.
I20260810 22:34:08.714660 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.66:0
--local_ip_for_outbound_sockets=127.16.233.66
--webserver_interface=127.16.233.66
--webserver_port=0
--tserver_master_addrs=127.16.233.126:45327
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab
--principal=kudu/127.16.233.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:40843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:34:08.850003 23691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:34:08.850330 23691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:34:08.850378 23691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:34:08.855036 23691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:34:08.855150 23691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:34:08.855218 23691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.66
I20260810 22:34:08.861368 23691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:40843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab
--principal=kudu/127.16.233.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.233.66
--webserver_port=0
--tserver_master_addrs=127.16.233.126:45327
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.66
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:34:08.862654 23691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:34:08.863813 23691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:34:08.874739 23699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:08.874749 23697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:08.874758 23696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:08.875070 23691 server_base.cc:1034] running on GCE node
I20260810 22:34:08.875721 23691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:34:08.876503 23691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:34:08.877741 23691 hybrid_clock.cc:648] HybridClock initialized: now 1786401248877700 us; error 62 us; skew 500 ppm
Aug 10 22:34:08 dist-test-slave-77v9 krb5kdc[23421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401248, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:34:08.881371 23691 init.cc:377] Logged in from keytab as kudu/127.16.233.66@KRBTEST.COM (short username kudu)
I20260810 22:34:08.882424 23691 webserver.cc:460] Webserver started at http://127.16.233.66:39801/ using document root <none> and password file <none>
I20260810 22:34:08.882988 23691 fs_manager.cc:359] Metadata directory not provided
I20260810 22:34:08.883051 23691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:34:08.883229 23691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:34:08.885701 23691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/minicluster-data/ts-1/data/instance:
uuid: "1edb89e163d74bad8cb2e49de1cf6fcf"
format_stamp: "Formatted at 2026-08-10 22:34:08 on dist-test-slave-77v9"
server_key: "0fcb5ef2fb560e7797aa1cec64959631"
server_key_iv: "9acde5ab75c531cbfbc44c35dba6da65"
server_key_version: "encryptionkey@0"
I20260810 22:34:08.886291 23691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/minicluster-data/ts-1/wal/instance:
uuid: "1edb89e163d74bad8cb2e49de1cf6fcf"
format_stamp: "Formatted at 2026-08-10 22:34:08 on dist-test-slave-77v9"
server_key: "0fcb5ef2fb560e7797aa1cec64959631"
server_key_iv: "9acde5ab75c531cbfbc44c35dba6da65"
server_key_version: "encryptionkey@0"
I20260810 22:34:08.890515 23691 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 22:34:08.893361 23708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:08.894901 23691 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 22:34:08.895015 23691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/minicluster-data/ts-1/wal
uuid: "1edb89e163d74bad8cb2e49de1cf6fcf"
format_stamp: "Formatted at 2026-08-10 22:34:08 on dist-test-slave-77v9"
server_key: "0fcb5ef2fb560e7797aa1cec64959631"
server_key_iv: "9acde5ab75c531cbfbc44c35dba6da65"
server_key_version: "encryptionkey@0"
I20260810 22:34:08.895125 23691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:34:08.915367 23691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:34:08.919121 23691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:34:08.919411 23691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:34:08.920258 23691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:34:08.921341 23691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:34:08.921416 23691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:08.921497 23691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:34:08.921543 23691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:08.932160 23691 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.66:40331
I20260810 22:34:08.932183 23833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.66:40331 every 8 connection(s)
I20260810 22:34:08.933354 23691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
I20260810 22:34:08.934617 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 23691
I20260810 22:34:08.934744 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/minicluster-data/ts-1/wal/instance
I20260810 22:34:08.935031 17317 external_mini_cluster.cc:1428] Setting key 25e174d8d17c245dbd8036c64ebfbc1b
WARNING: no policy specified for kudu/127.16.233.67@KRBTEST.COM; defaulting to no policy
Aug 10 22:34:08 dist-test-slave-77v9 krb5kdc[23421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401248, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.66@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.67@KRBTEST.COM" created.
I20260810 22:34:08.952329 23517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:08.935832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:50615 (local address 127.16.233.126:45327)
0810 22:34:08.936000 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:08.936004 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:08.937418 (+ 1414us) server_negotiation.cc:407] Connection header received
0810 22:34:08.939630 (+ 2212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:08.939636 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:08.939713 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:08.939925 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:08.941430 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:08.942376 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:08.944074 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:08.944522 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:08.944657 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:08.947638 (+ 2981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:08.947642 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:08.947644 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:08.947676 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:08.949419 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:08.950196 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:08.950199 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:08.950201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:08.950292 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:08.950740 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:08.950742 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:08.950743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:08.951054 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:08.951167 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:08.952039 (+ 872us) server_negotiation.cc:299] Negotiation successful
0810 22:34:08.952183 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260810 22:34:08.953717 23837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:08.936439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:45327 (local address 127.16.233.66:50615)
0810 22:34:08.937207 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:08.937277 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:08.939303 (+ 2026us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:08.940130 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:08.940138 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:08.940596 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:08.941243 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:08.941268 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:08.942565 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:08.942568 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:08.943905 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:08.943918 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:08.944721 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:08.944724 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:08.945046 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:08.945756 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:08.945775 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:08.947423 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:34:08.949562 (+ 2139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:08.949568 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:08.949584 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:08.950053 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:08.950416 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:08.950419 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:08.950420 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:08.950591 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:08.951218 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:34:08.951228 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:34:08.951645 (+ 417us) client_negotiation.cc:769] Sending connection context
0810 22:34:08.951907 (+ 262us) client_negotiation.cc:240] Negotiation successful
0810 22:34:08.952344 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":179,"threads_started":1}
I20260810 22:34:08.955443 23834 heartbeater.cc:347] Connected to a master server at 127.16.233.126:45327
I20260810 22:34:08.955739 23834 heartbeater.cc:464] Registering TS with master...
I20260810 22:34:08.956427 23834 heartbeater.cc:511] Master 127.16.233.126:45327 requested a full tablet report, sending...
I20260810 22:34:08.957906 23469 ts_manager.cc:194] Registered new tserver with Master: 1edb89e163d74bad8cb2e49de1cf6fcf (127.16.233.66:40331)
I20260810 22:34:08.958726 23469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.66@KRBTEST.COM'} at 127.16.233.66:50615
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab.
Entry for principal kudu/127.16.233.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab.
I20260810 22:34:08.974150 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.67:0
--local_ip_for_outbound_sockets=127.16.233.67
--webserver_interface=127.16.233.67
--webserver_port=0
--tserver_master_addrs=127.16.233.126:45327
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab
--principal=kudu/127.16.233.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:40843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:34:09.097198 23840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:34:09.097509 23840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:34:09.097555 23840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:34:09.102033 23840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:34:09.102116 23840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:34:09.102180 23840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.67
I20260810 22:34:09.107905 23840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:40843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab
--principal=kudu/127.16.233.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.233.67
--webserver_port=0
--tserver_master_addrs=127.16.233.126:45327
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.67
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:34:09.109189 23840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:34:09.110317 23840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:34:09.121294 23848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:09.121290 23850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:09.121495 23840 server_base.cc:1034] running on GCE node
W20260810 22:34:09.121685 23846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:09.122169 23840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:34:09.122946 23840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:34:09.124142 23840 hybrid_clock.cc:648] HybridClock initialized: now 1786401249124129 us; error 28 us; skew 500 ppm
Aug 10 22:34:09 dist-test-slave-77v9 krb5kdc[23421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401249, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:34:09.127298 23840 init.cc:377] Logged in from keytab as kudu/127.16.233.67@KRBTEST.COM (short username kudu)
I20260810 22:34:09.128317 23840 webserver.cc:460] Webserver started at http://127.16.233.67:34361/ using document root <none> and password file <none>
I20260810 22:34:09.129010 23840 fs_manager.cc:359] Metadata directory not provided
I20260810 22:34:09.129119 23840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:34:09.129372 23840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:34:09.131595 23840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/minicluster-data/ts-2/data/instance:
uuid: "b4912a9e35ab4f5886312f555172a9d4"
format_stamp: "Formatted at 2026-08-10 22:34:09 on dist-test-slave-77v9"
server_key: "fe2c5faef3dd09a93e37ff3add4d4664"
server_key_iv: "a76c7ba2263cebc51c22e5831e9d2bdc"
server_key_version: "encryptionkey@0"
I20260810 22:34:09.132339 23840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/minicluster-data/ts-2/wal/instance:
uuid: "b4912a9e35ab4f5886312f555172a9d4"
format_stamp: "Formatted at 2026-08-10 22:34:09 on dist-test-slave-77v9"
server_key: "fe2c5faef3dd09a93e37ff3add4d4664"
server_key_iv: "a76c7ba2263cebc51c22e5831e9d2bdc"
server_key_version: "encryptionkey@0"
I20260810 22:34:09.137137 23840 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.001s
I20260810 22:34:09.140126 23858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:09.141613 23840 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:34:09.141826 23840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/minicluster-data/ts-2/wal
uuid: "b4912a9e35ab4f5886312f555172a9d4"
format_stamp: "Formatted at 2026-08-10 22:34:09 on dist-test-slave-77v9"
server_key: "fe2c5faef3dd09a93e37ff3add4d4664"
server_key_iv: "a76c7ba2263cebc51c22e5831e9d2bdc"
server_key_version: "encryptionkey@0"
I20260810 22:34:09.141997 23840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:34:09.157047 23840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:34:09.161152 23840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:34:09.161405 23840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:34:09.162199 23840 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:34:09.163198 23840 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:34:09.163283 23840 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:09.163362 23840 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:34:09.163419 23840 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:09.174187 23840 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.67:35445
I20260810 22:34:09.174211 23981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.67:35445 every 8 connection(s)
I20260810 22:34:09.175272 23840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
I20260810 22:34:09.183787 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 23840
I20260810 22:34:09.183930 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/minicluster-data/ts-2/wal/instance
I20260810 22:34:09.184273 17317 external_mini_cluster.cc:1428] Setting key d4067584d9f72383141dd510f7676c4e
Aug 10 22:34:09 dist-test-slave-77v9 krb5kdc[23421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401249, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.67@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:34:09.191494 23517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:09.177736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:44693 (local address 127.16.233.126:45327)
0810 22:34:09.177983 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:09.177987 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:09.179067 (+ 1080us) server_negotiation.cc:407] Connection header received
0810 22:34:09.180471 (+ 1404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:09.180474 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:09.180527 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:09.180612 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:09.181947 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:09.182656 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:09.184212 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:09.184558 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:09.184654 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:09.187411 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:09.187415 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:09.187417 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:09.187445 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:09.188981 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:09.189625 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:09.189628 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:09.189630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:09.189839 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:09.190273 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:09.190277 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:09.190280 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:09.190532 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:09.190643 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:09.191202 (+ 559us) server_negotiation.cc:299] Negotiation successful
0810 22:34:09.191347 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139}
I20260810 22:34:09.192478 23984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:09.178277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:45327 (local address 127.16.233.67:44693)
0810 22:34:09.178889 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:09.178938 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:09.180234 (+ 1296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:09.180789 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:09.180796 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:09.181250 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:09.181760 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:09.181775 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:09.182809 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:09.182812 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:09.184073 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:09.184086 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:09.184721 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:09.184733 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:09.184957 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:09.185644 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:09.185662 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:09.187237 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:34:09.189100 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:09.189106 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:09.189120 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:09.189448 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:09.189985 (+ 537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:09.189988 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:09.189989 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:09.190156 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:09.190638 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:34:09.190643 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:34:09.190911 (+ 268us) client_negotiation.cc:769] Sending connection context
0810 22:34:09.191138 (+ 227us) client_negotiation.cc:240] Negotiation successful
0810 22:34:09.191448 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":147,"threads_started":1}
I20260810 22:34:09.193948 23982 heartbeater.cc:347] Connected to a master server at 127.16.233.126:45327
I20260810 22:34:09.194180 23982 heartbeater.cc:464] Registering TS with master...
I20260810 22:34:09.194669 23982 heartbeater.cc:511] Master 127.16.233.126:45327 requested a full tablet report, sending...
I20260810 22:34:09.195969 23469 ts_manager.cc:194] Registered new tserver with Master: b4912a9e35ab4f5886312f555172a9d4 (127.16.233.67:35445)
I20260810 22:34:09.196728 23469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.67@KRBTEST.COM'} at 127.16.233.67:44693
I20260810 22:34:09.198879 17317 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/krb5kdc/test-admin.keytab.
Aug 10 22:34:09 dist-test-slave-77v9 krb5kdc[23421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:34:09.219543 17317 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260810 22:34:09.223001 23990 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:34:09.221407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55734 (local address 127.16.233.65:34543)
0810 22:34:09.221804 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:09.221812 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:09.221842 (+ 30us) server_negotiation.cc:407] Connection header received
0810 22:34:09.221898 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:09.221903 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:09.222124 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:09.222275 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:09.222785 (+ 510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55734: BlockingRecv error: recv got EOF from 127.0.0.1:55734 (error 108)
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":155,"threads_started":1}
W20260810 22:34:09.223170 17317 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/krb5kdc/krb5cc' of type FILE
Aug 10 22:34:09 dist-test-slave-77v9 krb5kdc[23421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:34:09.224476 17317 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 22:34:09 dist-test-slave-77v9 krb5kdc[23421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:34:09.225797 17317 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 22:34:09 dist-test-slave-77v9 krb5kdc[23421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.65@KRBTEST.COM
I20260810 22:34:09.238790 23990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:09.227090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55740 (local address 127.16.233.65:34543)
0810 22:34:09.227273 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:09.227277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:09.227298 (+ 21us) server_negotiation.cc:407] Connection header received
0810 22:34:09.227441 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:09.227443 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:09.227492 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:09.227569 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:09.228475 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:09.229486 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:09.230544 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:09.231019 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:09.231125 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:09.234247 (+ 3122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:09.234262 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:09.234279 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:09.234314 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:09.236484 (+ 2170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:09.237200 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:09.237207 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:09.237208 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:09.237310 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:09.237637 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:09.237639 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:09.237640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:09.237972 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:09.238138 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:09.238391 (+ 253us) server_negotiation.cc:299] Negotiation successful
0810 22:34:09.238600 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
W20260810 22:34:09.243593 23990 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:34:09.242360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55756 (local address 127.16.233.65:34543)
0810 22:34:09.242503 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:09.242506 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:09.242683 (+ 177us) server_negotiation.cc:407] Connection header received
0810 22:34:09.242829 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:09.242832 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:09.242880 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:09.242945 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:09.243454 (+ 509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55756: BlockingRecv error: recv got EOF from 127.0.0.1:55756 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260810 22:34:09.243806 17317 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/krb5kdc/krb5cc' of type FILE
Aug 10 22:34:09 dist-test-slave-77v9 krb5kdc[23421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:34:09.245021 17317 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 22:34:09 dist-test-slave-77v9 krb5kdc[23421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:34:09.246037 17317 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 22:34:09 dist-test-slave-77v9 krb5kdc[23421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.65@KRBTEST.COM
I20260810 22:34:09.256598 23990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:09.247066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55764 (local address 127.16.233.65:34543)
0810 22:34:09.247198 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:09.247202 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:09.247414 (+ 212us) server_negotiation.cc:407] Connection header received
0810 22:34:09.247473 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:09.247476 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:09.247527 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:09.247618 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:09.248256 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:09.249130 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:09.249997 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:09.250218 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:09.250306 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:09.253095 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:09.253098 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:09.253100 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:09.253129 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:09.254847 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:09.255290 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:09.255293 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:09.255294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:09.255380 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:09.255662 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:09.255665 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:09.255666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:09.255941 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:09.256077 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:09.256343 (+ 266us) server_negotiation.cc:299] Negotiation successful
0810 22:34:09.256460 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260810 22:34:09.260294 23990 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:34:09.259000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55774 (local address 127.16.233.65:34543)
0810 22:34:09.259158 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:09.259163 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:09.259259 (+ 96us) server_negotiation.cc:407] Connection header received
0810 22:34:09.259438 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:09.259441 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:09.259497 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:09.259590 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:09.260157 (+ 567us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55774: BlockingRecv error: recv got EOF from 127.0.0.1:55774 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260810 22:34:09.260669 17317 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786401215402979-17317-0/krb5kdc/krb5cc)
Aug 10 22:34:09 dist-test-slave-77v9 krb5kdc[23421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:34:09.261837 17317 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 22:34:09 dist-test-slave-77v9 krb5kdc[23421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:34:09.262759 17317 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 22:34:09 dist-test-slave-77v9 krb5kdc[23421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.65@KRBTEST.COM
I20260810 22:34:09.273309 23990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:09.263690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55776 (local address 127.16.233.65:34543)
0810 22:34:09.263903 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:09.263907 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:09.264005 (+ 98us) server_negotiation.cc:407] Connection header received
0810 22:34:09.264110 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:09.264113 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:09.264159 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:09.264253 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:09.264762 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:09.265357 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:09.266263 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:09.266499 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:09.266595 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:09.269102 (+ 2507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:09.269107 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:09.269109 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:09.269179 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:09.271203 (+ 2024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:09.271725 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:09.271729 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:09.271731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:09.271835 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:09.272134 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:09.272139 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:09.272141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:09.272412 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:09.272595 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:09.272920 (+ 325us) server_negotiation.cc:299] Negotiation successful
0810 22:34:09.273075 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260810 22:34:09.274554 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 23542
I20260810 22:34:09.284024 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 23691
I20260810 22:34:09.292737 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 23840
I20260810 22:34:09.304622 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 23434
2026-08-10T22:34:09Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2546 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:34:09 dist-test-slave-77v9 krb5kdc[24006](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:34:09 dist-test-slave-77v9 krb5kdc[24006](info): set up 2 sockets
Aug 10 22:34:09 dist-test-slave-77v9 krb5kdc[24006](info): commencing operation
krb5kdc: starting...
W20260810 22:34:10.435433 17317 mini_kdc.cc:121] Time spent starting KDC: real 1.104s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Aug 10 22:34:10 dist-test-slave-77v9 krb5kdc[24006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:34:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:34:10Z Disabled control of system clock
WARNING: no policy specified for oryx/127.16.233.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.233.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.233.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/krb5kdc/oryx_127.16.233.126.keytab.
Entry for principal oryx/127.16.233.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/krb5kdc/oryx_127.16.233.126.keytab.
I20260810 22:34:10.584219 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:39177
--webserver_interface=127.16.233.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/krb5kdc/oryx_127.16.233.126.keytab
--principal=oryx/127.16.233.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:33445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.126:39177
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:34:10.706239 24020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:34:10.706612 24020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:34:10.706703 24020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:34:10.711144 24020 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:34:10.711266 24020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:34:10.711297 24020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:34:10.711324 24020 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:34:10.711352 24020 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:34:10.717234 24020 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:33445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.126:39177
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/krb5kdc/oryx_127.16.233.126.keytab
--principal=oryx/127.16.233.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:39177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:34:10.718540 24020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:34:10.719681 24020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:34:10.729332 24027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:10.729368 24026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:10.729628 24020 server_base.cc:1034] running on GCE node
W20260810 22:34:10.729398 24029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:10.730477 24020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:34:10.731748 24020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:34:10.732957 24020 hybrid_clock.cc:648] HybridClock initialized: now 1786401250732928 us; error 46 us; skew 500 ppm
Aug 10 22:34:10 dist-test-slave-77v9 krb5kdc[24006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401250, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:34:10.736196 24020 init.cc:377] Logged in from keytab as oryx/127.16.233.126@KRBTEST.COM (short username oryx)
I20260810 22:34:10.737354 24020 webserver.cc:460] Webserver started at http://127.16.233.126:36553/ using document root <none> and password file <none>
I20260810 22:34:10.738072 24020 fs_manager.cc:359] Metadata directory not provided
I20260810 22:34:10.738191 24020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:34:10.738453 24020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:34:10.740721 24020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/master-0/data/instance:
uuid: "61c3a2bed2604ebcb00d2c5647841a3c"
format_stamp: "Formatted at 2026-08-10 22:34:10 on dist-test-slave-77v9"
server_key: "404b19f8ed23794b62ac50914e6073b7"
server_key_iv: "7261323c8573713559457f85f57b13f8"
server_key_version: "encryptionkey@0"
I20260810 22:34:10.741472 24020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/master-0/wal/instance:
uuid: "61c3a2bed2604ebcb00d2c5647841a3c"
format_stamp: "Formatted at 2026-08-10 22:34:10 on dist-test-slave-77v9"
server_key: "404b19f8ed23794b62ac50914e6073b7"
server_key_iv: "7261323c8573713559457f85f57b13f8"
server_key_version: "encryptionkey@0"
I20260810 22:34:10.746062 24020 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 22:34:10.749130 24037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:10.750662 24020 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:34:10.750937 24020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/master-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/master-0/wal
uuid: "61c3a2bed2604ebcb00d2c5647841a3c"
format_stamp: "Formatted at 2026-08-10 22:34:10 on dist-test-slave-77v9"
server_key: "404b19f8ed23794b62ac50914e6073b7"
server_key_iv: "7261323c8573713559457f85f57b13f8"
server_key_version: "encryptionkey@0"
I20260810 22:34:10.751098 24020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:34:10.767385 24020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:34:10.771504 24020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:34:10.771782 24020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:34:10.781272 24020 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.126:39177
I20260810 22:34:10.781705 24094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.126:39177 every 8 connection(s)
I20260810 22:34:10.783010 24020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
I20260810 22:34:10.783778 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 24020
I20260810 22:34:10.783910 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/master-0/wal/instance
I20260810 22:34:10.784236 17317 external_mini_cluster.cc:1428] Setting key 6a6133d2c709536148867abb644a599d
I20260810 22:34:10.786728 24095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:34:10.794232 24095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61c3a2bed2604ebcb00d2c5647841a3c: Bootstrap starting.
Aug 10 22:34:10 dist-test-slave-77v9 krb5kdc[24006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM
I20260810 22:34:10.797446 24095 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61c3a2bed2604ebcb00d2c5647841a3c: Neither blocks nor log segments found. Creating new log.
I20260810 22:34:10.798554 24095 log.cc:824] T 00000000000000000000000000000000 P 61c3a2bed2604ebcb00d2c5647841a3c: Log is configured to *not* fsync() on all Append() calls
I20260810 22:34:10.801566 24095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61c3a2bed2604ebcb00d2c5647841a3c: No bootstrap required, opened a new log
I20260810 22:34:10.802546 24098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:10.785918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45586 (local address 127.16.233.126:39177)
0810 22:34:10.786416 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:10.786438 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:10.786500 (+ 62us) server_negotiation.cc:407] Connection header received
0810 22:34:10.787386 (+ 886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:10.787410 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:10.787894 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:10.788305 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:10.789339 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:10.790906 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:10.791936 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:10.792657 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:10.792882 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:10.796037 (+ 3155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:10.796044 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:10.796062 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:10.796103 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:10.798688 (+ 2585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:10.799405 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:10.799411 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:10.799417 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:10.799544 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:10.799953 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:10.799957 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:10.799959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:10.800548 (+ 589us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:10.800758 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:10.801175 (+ 417us) server_negotiation.cc:299] Negotiation successful
0810 22:34:10.801511 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":140,"threads_started":1}
I20260810 22:34:10.806236 24095 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61c3a2bed2604ebcb00d2c5647841a3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39177 } }
I20260810 22:34:10.806528 24095 raft_consensus.cc:374] T 00000000000000000000000000000000 P 61c3a2bed2604ebcb00d2c5647841a3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:34:10.806579 24095 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61c3a2bed2604ebcb00d2c5647841a3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61c3a2bed2604ebcb00d2c5647841a3c, State: Initialized, Role: FOLLOWER
I20260810 22:34:10.807022 24095 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61c3a2bed2604ebcb00d2c5647841a3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39177 } }
I20260810 22:34:10.807168 24095 raft_consensus.cc:388] T 00000000000000000000000000000000 P 61c3a2bed2604ebcb00d2c5647841a3c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:34:10.807211 24095 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61c3a2bed2604ebcb00d2c5647841a3c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:34:10.807287 24095 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61c3a2bed2604ebcb00d2c5647841a3c [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:34:10.808365 24095 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61c3a2bed2604ebcb00d2c5647841a3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39177 } }
I20260810 22:34:10.808768 24095 leader_election.cc:302] T 00000000000000000000000000000000 P 61c3a2bed2604ebcb00d2c5647841a3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61c3a2bed2604ebcb00d2c5647841a3c; no voters:
I20260810 22:34:10.809242 24095 leader_election.cc:288] T 00000000000000000000000000000000 P 61c3a2bed2604ebcb00d2c5647841a3c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:34:10.809346 24101 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61c3a2bed2604ebcb00d2c5647841a3c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:34:10.809540 24101 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61c3a2bed2604ebcb00d2c5647841a3c [term 1 LEADER]: Becoming Leader. State: Replica: 61c3a2bed2604ebcb00d2c5647841a3c, State: Running, Role: LEADER
I20260810 22:34:10.809863 24101 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61c3a2bed2604ebcb00d2c5647841a3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39177 } }
I20260810 22:34:10.810705 24095 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61c3a2bed2604ebcb00d2c5647841a3c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:34:10.811765 24102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61c3a2bed2604ebcb00d2c5647841a3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61c3a2bed2604ebcb00d2c5647841a3c. Latest consensus state: current_term: 1 leader_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39177 } } }
I20260810 22:34:10.811996 24102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61c3a2bed2604ebcb00d2c5647841a3c [sys.catalog]: This master's current role is: LEADER
I20260810 22:34:10.812250 24103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61c3a2bed2604ebcb00d2c5647841a3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39177 } } }
I20260810 22:34:10.812359 24103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61c3a2bed2604ebcb00d2c5647841a3c [sys.catalog]: This master's current role is: LEADER
I20260810 22:34:10.812481 24112 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:34:10.816280 24112 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:34:10.823026 24112 catalog_manager.cc:1347] Generated new cluster ID: 0d14ee0609a54fd5bfab4629910fd236
I20260810 22:34:10.823140 24112 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:34:10.831001 24112 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:34:10.832283 24112 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:34:10.846087 24112 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 61c3a2bed2604ebcb00d2c5647841a3c: Generated new TSK 0
I20260810 22:34:10.847172 24112 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.16.233.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.233.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.233.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/krb5kdc/oryx_127.16.233.65.keytab.
Entry for principal oryx/127.16.233.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/krb5kdc/oryx_127.16.233.65.keytab.
I20260810 22:34:10.898386 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.65:0
--local_ip_for_outbound_sockets=127.16.233.65
--webserver_interface=127.16.233.65
--webserver_port=0
--tserver_master_addrs=127.16.233.126:39177
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/krb5kdc/oryx_127.16.233.65.keytab
--principal=oryx/127.16.233.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:33445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:34:11.026696 24124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:34:11.027067 24124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:34:11.027159 24124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:34:11.031599 24124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:34:11.031725 24124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:34:11.031832 24124 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.65
I20260810 22:34:11.037727 24124 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:33445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/krb5kdc/oryx_127.16.233.65.keytab
--principal=oryx/127.16.233.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.233.65
--webserver_port=0
--tserver_master_addrs=127.16.233.126:39177
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.65
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:34:11.039053 24124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:34:11.040283 24124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:34:11.051386 24129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:11.051359 24130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:11.051347 24133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:11.051656 24124 server_base.cc:1034] running on GCE node
I20260810 22:34:11.052153 24124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:34:11.052901 24124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:34:11.054147 24124 hybrid_clock.cc:648] HybridClock initialized: now 1786401251054095 us; error 73 us; skew 500 ppm
Aug 10 22:34:11 dist-test-slave-77v9 krb5kdc[24006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401251, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:34:11.057852 24124 init.cc:377] Logged in from keytab as oryx/127.16.233.65@KRBTEST.COM (short username oryx)
I20260810 22:34:11.058935 24124 webserver.cc:460] Webserver started at http://127.16.233.65:46001/ using document root <none> and password file <none>
I20260810 22:34:11.059615 24124 fs_manager.cc:359] Metadata directory not provided
I20260810 22:34:11.059725 24124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:34:11.059989 24124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:34:11.062440 24124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-0/data/instance:
uuid: "599ede616cdd412fa79af64abd77799a"
format_stamp: "Formatted at 2026-08-10 22:34:11 on dist-test-slave-77v9"
server_key: "5da5dac1fd1333489c640283d02ce6e4"
server_key_iv: "5a93e3d275647d85d6f4df79b70e202e"
server_key_version: "encryptionkey@0"
I20260810 22:34:11.063105 24124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-0/wal/instance:
uuid: "599ede616cdd412fa79af64abd77799a"
format_stamp: "Formatted at 2026-08-10 22:34:11 on dist-test-slave-77v9"
server_key: "5da5dac1fd1333489c640283d02ce6e4"
server_key_iv: "5a93e3d275647d85d6f4df79b70e202e"
server_key_version: "encryptionkey@0"
I20260810 22:34:11.067582 24124 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 22:34:11.070636 24145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:11.072451 24124 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 22:34:11.072641 24124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-0/wal
uuid: "599ede616cdd412fa79af64abd77799a"
format_stamp: "Formatted at 2026-08-10 22:34:11 on dist-test-slave-77v9"
server_key: "5da5dac1fd1333489c640283d02ce6e4"
server_key_iv: "5a93e3d275647d85d6f4df79b70e202e"
server_key_version: "encryptionkey@0"
I20260810 22:34:11.072788 24124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:34:11.095965 24124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:34:11.100266 24124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:34:11.100591 24124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:34:11.101439 24124 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:34:11.102560 24124 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:34:11.102654 24124 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:11.102742 24124 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:34:11.102793 24124 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:11.114923 24124 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.65:39951
I20260810 22:34:11.114979 24267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.65:39951 every 8 connection(s)
I20260810 22:34:11.116091 24124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
I20260810 22:34:11.118319 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 24124
I20260810 22:34:11.118455 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-0/wal/instance
I20260810 22:34:11.118772 17317 external_mini_cluster.cc:1428] Setting key 778ff0ebd7391962b64e28a9fa06ccce
Aug 10 22:34:11 dist-test-slave-77v9 krb5kdc[24006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401251, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.65@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM
WARNING: no policy specified for oryx/127.16.233.66@KRBTEST.COM; defaulting to no policy
I20260810 22:34:11.133035 24098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:11.118479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:48027 (local address 127.16.233.126:39177)
0810 22:34:11.118638 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:11.118642 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:11.119708 (+ 1066us) server_negotiation.cc:407] Connection header received
0810 22:34:11.121319 (+ 1611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:11.121323 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:11.121381 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:11.121505 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:11.122925 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:11.123516 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:11.124965 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:11.125225 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:11.125349 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:11.128476 (+ 3127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:11.128479 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:11.128481 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:11.128510 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:11.130579 (+ 2069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:11.131281 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:11.131284 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:11.131285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:11.131368 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:11.131764 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:11.131768 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:11.131769 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:11.131992 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:11.132105 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:11.132665 (+ 560us) server_negotiation.cc:299] Negotiation successful
0810 22:34:11.132857 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260810 22:34:11.133967 24270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:11.118916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39177 (local address 127.16.233.65:48027)
0810 22:34:11.119542 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:11.119603 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:11.121078 (+ 1475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:11.121703 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:11.121711 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:11.122117 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:11.122728 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:11.122754 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:11.123677 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:11.123683 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:11.124777 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:11.124787 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:11.125408 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:11.125413 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:11.125609 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:11.126407 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:11.126425 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:11.128248 (+ 1823us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:34:11.130724 (+ 2476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:11.130812 (+ 88us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:11.130829 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:11.131131 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:11.131468 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:11.131471 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:11.131473 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:11.131648 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:11.132106 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:34:11.132114 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:34:11.132410 (+ 296us) client_negotiation.cc:769] Sending connection context
0810 22:34:11.132622 (+ 212us) client_negotiation.cc:240] Negotiation successful
0810 22:34:11.133054 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":210,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.233.66@KRBTEST.COM" created.
I20260810 22:34:11.135720 24268 heartbeater.cc:347] Connected to a master server at 127.16.233.126:39177
I20260810 22:34:11.136030 24268 heartbeater.cc:464] Registering TS with master...
I20260810 22:34:11.136603 24268 heartbeater.cc:511] Master 127.16.233.126:39177 requested a full tablet report, sending...
I20260810 22:34:11.138630 24057 ts_manager.cc:194] Registered new tserver with Master: 599ede616cdd412fa79af64abd77799a (127.16.233.65:39951)
I20260810 22:34:11.140391 24057 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.233.65@KRBTEST.COM'} at 127.16.233.65:48027
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.233.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/krb5kdc/oryx_127.16.233.66.keytab.
Entry for principal oryx/127.16.233.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/krb5kdc/oryx_127.16.233.66.keytab.
I20260810 22:34:11.158133 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.66:0
--local_ip_for_outbound_sockets=127.16.233.66
--webserver_interface=127.16.233.66
--webserver_port=0
--tserver_master_addrs=127.16.233.126:39177
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/krb5kdc/oryx_127.16.233.66.keytab
--principal=oryx/127.16.233.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:33445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:34:11.280352 24273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:34:11.280660 24273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:34:11.280706 24273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:34:11.285120 24273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:34:11.285220 24273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:34:11.285295 24273 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.66
I20260810 22:34:11.291585 24273 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:33445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/krb5kdc/oryx_127.16.233.66.keytab
--principal=oryx/127.16.233.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.233.66
--webserver_port=0
--tserver_master_addrs=127.16.233.126:39177
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24273
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.66
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:34:11.292922 24273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:34:11.294071 24273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:34:11.305438 24279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:11.305394 24283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:11.305395 24278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:11.305855 24273 server_base.cc:1034] running on GCE node
I20260810 22:34:11.306412 24273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:34:11.307247 24273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:34:11.308462 24273 hybrid_clock.cc:648] HybridClock initialized: now 1786401251308425 us; error 56 us; skew 500 ppm
Aug 10 22:34:11 dist-test-slave-77v9 krb5kdc[24006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401251, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:34:11.313187 24273 init.cc:377] Logged in from keytab as oryx/127.16.233.66@KRBTEST.COM (short username oryx)
I20260810 22:34:11.314812 24273 webserver.cc:460] Webserver started at http://127.16.233.66:39831/ using document root <none> and password file <none>
I20260810 22:34:11.315672 24273 fs_manager.cc:359] Metadata directory not provided
I20260810 22:34:11.315752 24273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:34:11.316043 24273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:34:11.318743 24273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-1/data/instance:
uuid: "83e9b7216a8040b3a3418e588ecf2959"
format_stamp: "Formatted at 2026-08-10 22:34:11 on dist-test-slave-77v9"
server_key: "ddf5a5a39f27720ceef86b21a0bdbbb0"
server_key_iv: "2c7b95f0aed390311bce3452a0031e4e"
server_key_version: "encryptionkey@0"
I20260810 22:34:11.319459 24273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-1/wal/instance:
uuid: "83e9b7216a8040b3a3418e588ecf2959"
format_stamp: "Formatted at 2026-08-10 22:34:11 on dist-test-slave-77v9"
server_key: "ddf5a5a39f27720ceef86b21a0bdbbb0"
server_key_iv: "2c7b95f0aed390311bce3452a0031e4e"
server_key_version: "encryptionkey@0"
I20260810 22:34:11.324394 24273 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260810 22:34:11.327826 24291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:11.329622 24273 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:34:11.329794 24273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-1/wal
uuid: "83e9b7216a8040b3a3418e588ecf2959"
format_stamp: "Formatted at 2026-08-10 22:34:11 on dist-test-slave-77v9"
server_key: "ddf5a5a39f27720ceef86b21a0bdbbb0"
server_key_iv: "2c7b95f0aed390311bce3452a0031e4e"
server_key_version: "encryptionkey@0"
I20260810 22:34:11.329946 24273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:34:11.344674 24273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:34:11.349372 24273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:34:11.349638 24273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:34:11.350356 24273 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:34:11.351294 24273 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:34:11.351380 24273 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:11.351454 24273 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:34:11.351500 24273 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:11.363174 24273 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.66:37385
I20260810 22:34:11.363242 24413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.66:37385 every 8 connection(s)
I20260810 22:34:11.364266 24273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
I20260810 22:34:11.367623 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 24273
I20260810 22:34:11.367759 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-1/wal/instance
I20260810 22:34:11.368088 17317 external_mini_cluster.cc:1428] Setting key f7df8f89b50d5826c4d2410b8a97919a
Aug 10 22:34:11 dist-test-slave-77v9 krb5kdc[24006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401251, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.66@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM
WARNING: no policy specified for oryx/127.16.233.67@KRBTEST.COM; defaulting to no policy
I20260810 22:34:11.381392 24098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:11.366671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:55237 (local address 127.16.233.126:39177)
0810 22:34:11.366932 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:11.366938 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:11.367922 (+ 984us) server_negotiation.cc:407] Connection header received
0810 22:34:11.369755 (+ 1833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:11.369761 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:11.369822 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:11.369909 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:11.371312 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:11.371890 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:11.373357 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:11.373608 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:11.373744 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:11.376802 (+ 3058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:11.376836 (+ 34us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:11.376842 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:11.376888 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:11.378720 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:11.379387 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:11.379391 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:11.379393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:11.379499 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:11.379931 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:11.379936 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:11.379938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:11.380174 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:11.380304 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:11.380922 (+ 618us) server_negotiation.cc:299] Negotiation successful
0810 22:34:11.381193 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145}
I20260810 22:34:11.382442 24416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:11.367146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39177 (local address 127.16.233.66:55237)
0810 22:34:11.367731 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:11.367790 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:11.369449 (+ 1659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:11.370093 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:11.370100 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:11.370507 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:11.371127 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:11.371143 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:11.372050 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:11.372055 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:11.373136 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:11.373148 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:11.373799 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:11.373803 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:11.374002 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:11.374727 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:11.374746 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:11.376556 (+ 1810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:34:11.378883 (+ 2327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:11.378890 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:11.378910 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:11.379251 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:11.379620 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:11.379625 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:11.379627 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:11.379815 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:11.380302 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:34:11.380310 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:34:11.380609 (+ 299us) client_negotiation.cc:769] Sending connection context
0810 22:34:11.380859 (+ 250us) client_negotiation.cc:240] Negotiation successful
0810 22:34:11.381268 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":184,"threads_started":1}
I20260810 22:34:11.384014 24414 heartbeater.cc:347] Connected to a master server at 127.16.233.126:39177
I20260810 22:34:11.384330 24414 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.233.67@KRBTEST.COM" created.
I20260810 22:34:11.385105 24414 heartbeater.cc:511] Master 127.16.233.126:39177 requested a full tablet report, sending...
I20260810 22:34:11.386653 24057 ts_manager.cc:194] Registered new tserver with Master: 83e9b7216a8040b3a3418e588ecf2959 (127.16.233.66:37385)
I20260810 22:34:11.387557 24057 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.233.66@KRBTEST.COM'} at 127.16.233.66:55237
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.233.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/krb5kdc/oryx_127.16.233.67.keytab.
Entry for principal oryx/127.16.233.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/krb5kdc/oryx_127.16.233.67.keytab.
I20260810 22:34:11.409956 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.67:0
--local_ip_for_outbound_sockets=127.16.233.67
--webserver_interface=127.16.233.67
--webserver_port=0
--tserver_master_addrs=127.16.233.126:39177
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/krb5kdc/oryx_127.16.233.67.keytab
--principal=oryx/127.16.233.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:33445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:34:11.539717 24419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:34:11.540124 24419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:34:11.540177 24419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:34:11.544750 24419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:34:11.544937 24419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:34:11.545025 24419 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.67
I20260810 22:34:11.550932 24419 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:33445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/krb5kdc/oryx_127.16.233.67.keytab
--principal=oryx/127.16.233.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.233.67
--webserver_port=0
--tserver_master_addrs=127.16.233.126:39177
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.67
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:34:11.552145 24419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:34:11.553330 24419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:34:11.564502 24427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:11.564450 24424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:11.564469 24425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:11.564939 24419 server_base.cc:1034] running on GCE node
I20260810 22:34:11.565524 24419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:34:11.566318 24419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:34:11.567541 24419 hybrid_clock.cc:648] HybridClock initialized: now 1786401251567522 us; error 35 us; skew 500 ppm
Aug 10 22:34:11 dist-test-slave-77v9 krb5kdc[24006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401251, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:34:11.570775 24419 init.cc:377] Logged in from keytab as oryx/127.16.233.67@KRBTEST.COM (short username oryx)
I20260810 22:34:11.571842 24419 webserver.cc:460] Webserver started at http://127.16.233.67:39629/ using document root <none> and password file <none>
I20260810 22:34:11.572428 24419 fs_manager.cc:359] Metadata directory not provided
I20260810 22:34:11.572496 24419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:34:11.572701 24419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:34:11.575114 24419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-2/data/instance:
uuid: "14098e2fbdf1420da6d97dd9a26616f1"
format_stamp: "Formatted at 2026-08-10 22:34:11 on dist-test-slave-77v9"
server_key: "dc6bd56a3e584d9d2a34b65bf71d2fa1"
server_key_iv: "85eee8965c601fc6a87336955fd464af"
server_key_version: "encryptionkey@0"
I20260810 22:34:11.575699 24419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-2/wal/instance:
uuid: "14098e2fbdf1420da6d97dd9a26616f1"
format_stamp: "Formatted at 2026-08-10 22:34:11 on dist-test-slave-77v9"
server_key: "dc6bd56a3e584d9d2a34b65bf71d2fa1"
server_key_iv: "85eee8965c601fc6a87336955fd464af"
server_key_version: "encryptionkey@0"
I20260810 22:34:11.580252 24419 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260810 22:34:11.583130 24436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:11.584585 24419 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 22:34:11.584703 24419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-2/wal
uuid: "14098e2fbdf1420da6d97dd9a26616f1"
format_stamp: "Formatted at 2026-08-10 22:34:11 on dist-test-slave-77v9"
server_key: "dc6bd56a3e584d9d2a34b65bf71d2fa1"
server_key_iv: "85eee8965c601fc6a87336955fd464af"
server_key_version: "encryptionkey@0"
I20260810 22:34:11.584906 24419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:34:11.615784 24419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:34:11.619540 24419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:34:11.619716 24419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:34:11.620314 24419 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:34:11.621342 24419 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:34:11.621403 24419 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:11.621446 24419 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:34:11.621461 24419 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:11.631211 24419 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.67:39107
I20260810 22:34:11.631229 24561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.67:39107 every 8 connection(s)
I20260810 22:34:11.632256 24419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
I20260810 22:34:11.641161 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 24419
I20260810 22:34:11.641314 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-2/wal/instance
I20260810 22:34:11.641644 17317 external_mini_cluster.cc:1428] Setting key f641ff40147267b7001e9c71dd37058b
Aug 10 22:34:11 dist-test-slave-77v9 krb5kdc[24006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401251, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.67@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM
I20260810 22:34:11.648138 24098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:11.634528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:55963 (local address 127.16.233.126:39177)
0810 22:34:11.634678 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:11.634682 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:11.635797 (+ 1115us) server_negotiation.cc:407] Connection header received
0810 22:34:11.637227 (+ 1430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:11.637230 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:11.637285 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:11.637376 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:11.638763 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:11.639314 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:11.640613 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:11.641014 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:11.641142 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:11.644264 (+ 3122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:11.644270 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:11.644273 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:11.644311 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:11.645900 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:11.646508 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:11.646516 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:11.646517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:11.646597 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:11.646969 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:11.646972 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:11.646974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:11.647176 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:11.647286 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:11.647833 (+ 547us) server_negotiation.cc:299] Negotiation successful
0810 22:34:11.647973 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 22:34:11.649094 24564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:11.635061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39177 (local address 127.16.233.67:55963)
0810 22:34:11.635611 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:11.635660 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:11.637004 (+ 1344us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:11.637592 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:11.637600 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:11.638038 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:11.638591 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:11.638605 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:11.639468 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:11.639471 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:11.640482 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:11.640493 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:11.641214 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:11.641217 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:11.641478 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:11.642207 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:11.642236 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:11.644056 (+ 1820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:34:11.646048 (+ 1992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:11.646055 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:11.646073 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:11.646372 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:11.646694 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:11.646698 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:11.646700 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:11.646861 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:11.647294 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:34:11.647304 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:34:11.647581 (+ 277us) client_negotiation.cc:769] Sending connection context
0810 22:34:11.647783 (+ 202us) client_negotiation.cc:240] Negotiation successful
0810 22:34:11.648144 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":126,"threads_started":1}
I20260810 22:34:11.650348 24562 heartbeater.cc:347] Connected to a master server at 127.16.233.126:39177
I20260810 22:34:11.650622 24562 heartbeater.cc:464] Registering TS with master...
I20260810 22:34:11.651168 24562 heartbeater.cc:511] Master 127.16.233.126:39177 requested a full tablet report, sending...
I20260810 22:34:11.652424 24057 ts_manager.cc:194] Registered new tserver with Master: 14098e2fbdf1420da6d97dd9a26616f1 (127.16.233.67:39107)
I20260810 22:34:11.653178 24057 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.233.67@KRBTEST.COM'} at 127.16.233.67:55963
I20260810 22:34:11.656411 17317 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.16.233.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.233.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.233.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/krb5kdc/oryx_127.16.233.125.keytab.
Entry for principal oryx/127.16.233.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/krb5kdc/oryx_127.16.233.125.keytab.
I20260810 22:34:11.693962 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.125:34439
--webserver_interface=127.16.233.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/krb5kdc/oryx_127.16.233.125.keytab
--principal=oryx/127.16.233.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:33445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.126:39177,127.16.233.125:34439
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:34:11.820680 24567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:34:11.821048 24567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:34:11.821094 24567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:34:11.825275 24567 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:34:11.825361 24567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:34:11.825384 24567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:34:11.825404 24567 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:34:11.825423 24567 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:34:11.831367 24567 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:33445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.126:39177,127.16.233.125:34439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/krb5kdc/oryx_127.16.233.125.keytab
--principal=oryx/127.16.233.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.125:34439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.233.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
W20260810 22:34:11.831686 24567 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.16.233.126:39177,127.16.233.125:34439'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260810 22:34:11.832756 24567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:34:11.834010 24567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:34:11.843758 24574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:11.843684 24577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:11.843684 24573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:11.843839 24567 server_base.cc:1034] running on GCE node
I20260810 22:34:11.844732 24567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:34:11.846101 24567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:34:11.847321 24567 hybrid_clock.cc:648] HybridClock initialized: now 1786401251847290 us; error 44 us; skew 500 ppm
Aug 10 22:34:11 dist-test-slave-77v9 krb5kdc[24006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401251, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:34:11.850891 24567 init.cc:377] Logged in from keytab as oryx/127.16.233.125@KRBTEST.COM (short username oryx)
I20260810 22:34:11.851946 24567 webserver.cc:460] Webserver started at http://127.16.233.125:41033/ using document root <none> and password file <none>
I20260810 22:34:11.852684 24567 fs_manager.cc:359] Metadata directory not provided
I20260810 22:34:11.852766 24567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:34:11.853044 24567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:34:11.855724 24567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/master-1/data/instance:
uuid: "580d354da5eb4d03a86d298eb6bc5af7"
format_stamp: "Formatted at 2026-08-10 22:34:11 on dist-test-slave-77v9"
server_key: "12d392cdda1221b99b8ee637a03f163d"
server_key_iv: "b51bb565f9e3cf76bbe1022502bf26eb"
server_key_version: "encryptionkey@0"
I20260810 22:34:11.856454 24567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/master-1/wal/instance:
uuid: "580d354da5eb4d03a86d298eb6bc5af7"
format_stamp: "Formatted at 2026-08-10 22:34:11 on dist-test-slave-77v9"
server_key: "12d392cdda1221b99b8ee637a03f163d"
server_key_iv: "b51bb565f9e3cf76bbe1022502bf26eb"
server_key_version: "encryptionkey@0"
I20260810 22:34:11.860948 24567 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.001s
I20260810 22:34:11.863907 24584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:11.865532 24567 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260810 22:34:11.865708 24567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/master-1/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/master-1/wal
uuid: "580d354da5eb4d03a86d298eb6bc5af7"
format_stamp: "Formatted at 2026-08-10 22:34:11 on dist-test-slave-77v9"
server_key: "12d392cdda1221b99b8ee637a03f163d"
server_key_iv: "b51bb565f9e3cf76bbe1022502bf26eb"
server_key_version: "encryptionkey@0"
I20260810 22:34:11.865857 24567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:34:11.880563 24567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:34:11.884553 24567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:34:11.884852 24567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:34:11.893419 24567 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.125:34439
I20260810 22:34:11.893826 24646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.125:34439 every 8 connection(s)
I20260810 22:34:11.894762 24567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/master-1/data/info.pb
I20260810 22:34:11.897877 24647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:34:11.902316 24647 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39177 } has no permanent_uuid. Determining permanent_uuid...
I20260810 22:34:11.903956 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 24567
Aug 10 22:34:11 dist-test-slave-77v9 krb5kdc[24006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401251, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.125@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM
Aug 10 22:34:11 dist-test-slave-77v9 krb5kdc[24006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.233.125@KRBTEST.COM
I20260810 22:34:11.918424 24098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:11.904912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45596 (local address 127.16.233.126:39177)
0810 22:34:11.905115 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:11.905120 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:11.906050 (+ 930us) server_negotiation.cc:407] Connection header received
0810 22:34:11.907288 (+ 1238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:11.907291 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:11.907346 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:11.907495 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:11.908440 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:11.909166 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:11.910341 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:11.910667 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:11.910794 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:11.914112 (+ 3318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:11.914116 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:11.914126 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:11.914155 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:11.915957 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:11.916640 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:11.916643 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:11.916644 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:11.916722 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:11.917223 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:11.917227 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:11.917229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:11.917480 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:11.917598 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:11.918124 (+ 526us) server_negotiation.cc:299] Negotiation successful
0810 22:34:11.918251 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260810 22:34:11.919378 24656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:11.906137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50900 (local address 127.16.233.125:34439)
0810 22:34:11.906409 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:11.906420 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:11.906475 (+ 55us) server_negotiation.cc:407] Connection header received
0810 22:34:11.907002 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:11.907027 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:11.907294 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:11.907486 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:11.908381 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:11.909370 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:11.910282 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:11.910756 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:11.910953 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:11.914589 (+ 3636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:11.914596 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:11.914607 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:11.914630 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:11.916638 (+ 2008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:11.917167 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:11.917175 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:11.917177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:11.917267 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:11.917567 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:11.917570 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:11.917572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:11.917905 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:11.918117 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:11.918374 (+ 257us) server_negotiation.cc:299] Negotiation successful
0810 22:34:11.918526 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":70,"server-negotiator.queue_time_us":185,"spinlock_wait_cycles":6016,"thread_start_us":83,"threads_started":1}
I20260810 22:34:11.919380 24651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:11.905332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39177 (local address 127.0.0.1:45596)
0810 22:34:11.905898 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:11.905946 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:11.907002 (+ 1056us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:11.907554 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:11.907560 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:11.907994 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:11.908270 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:11.908284 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:11.909340 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:11.909343 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:11.910192 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:11.910202 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:11.910837 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:11.910841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:11.910998 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:11.912051 (+ 1053us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:11.912069 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:11.913931 (+ 1862us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:34:11.916076 (+ 2145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:11.916084 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:11.916095 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:11.916526 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:11.916953 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:11.916956 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:11.916958 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:11.917116 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:11.917597 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:34:11.917604 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:34:11.917896 (+ 292us) client_negotiation.cc:769] Sending connection context
0810 22:34:11.918080 (+ 184us) client_negotiation.cc:240] Negotiation successful
0810 22:34:11.918425 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":25,"spinlock_wait_cycles":6400,"thread_start_us":137,"threads_started":1}
I20260810 22:34:11.921125 24647 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.233.125" port: 34439 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 22:34:11 dist-test-slave-77v9 krb5kdc[24006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401251, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.125@KRBTEST.COM for oryx/127.16.233.125@KRBTEST.COM
I20260810 22:34:11.930241 24651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:11.921778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.125:34439 (local address 127.0.0.1:50910)
0810 22:34:11.921903 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:11.921927 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:11.922033 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:11.922350 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:11.922353 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:11.922513 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:11.922613 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:11.922618 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:11.923498 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:11.923501 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:11.924070 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:11.924079 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:11.924612 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:11.924615 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:11.924697 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:11.925287 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:11.925299 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:11.926663 (+ 1364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:34:11.928662 (+ 1999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:11.928665 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:11.928666 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:11.928964 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:11.929266 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:11.929268 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:11.929269 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:11.929349 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:11.929750 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:34:11.929753 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:34:11.929843 (+ 90us) client_negotiation.cc:769] Sending connection context
0810 22:34:11.929966 (+ 123us) client_negotiation.cc:240] Negotiation successful
0810 22:34:11.930104 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260810 22:34:11.930461 24656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:11.921943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50910 (local address 127.16.233.125:34439)
0810 22:34:11.922081 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:11.922088 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:11.922104 (+ 16us) server_negotiation.cc:407] Connection header received
0810 22:34:11.922138 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:11.922140 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:11.922201 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:11.922313 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:11.922806 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:11.923350 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:11.924205 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:11.924487 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:11.924584 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:11.926858 (+ 2274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:11.926861 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:11.926863 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:11.926884 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:11.928499 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:11.929076 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:11.929079 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:11.929080 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:11.929174 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:11.929436 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:11.929438 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:11.929439 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:11.929632 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:11.929715 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:11.930078 (+ 363us) server_negotiation.cc:299] Negotiation successful
0810 22:34:11.930181 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 22:34:11.934397 24647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7: Bootstrap starting.
I20260810 22:34:11.936887 24647 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7: Neither blocks nor log segments found. Creating new log.
I20260810 22:34:11.937657 24647 log.cc:824] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7: Log is configured to *not* fsync() on all Append() calls
I20260810 22:34:11.939927 24647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7: No bootstrap required, opened a new log
I20260810 22:34:11.943110 24647 raft_consensus.cc:348] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39177 } } peers { permanent_uuid: "580d354da5eb4d03a86d298eb6bc5af7" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 34439 } }
I20260810 22:34:11.943372 24647 raft_consensus.cc:374] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:34:11.943470 24647 raft_consensus.cc:729] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 580d354da5eb4d03a86d298eb6bc5af7, State: Initialized, Role: FOLLOWER
I20260810 22:34:11.943966 24647 consensus_queue.cc:260] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39177 } } peers { permanent_uuid: "580d354da5eb4d03a86d298eb6bc5af7" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 34439 } }
I20260810 22:34:11.944496 24659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39177 } } peers { permanent_uuid: "580d354da5eb4d03a86d298eb6bc5af7" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 34439 } } }
I20260810 22:34:11.944618 24659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 22:34:11.945339 24647 sys_catalog.cc:564] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7 [sys.catalog]: configured and running, proceeding with master startup.
W20260810 22:34:11.950507 24670 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 22:34:11.950619 24670 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 22:34:11.953729 24567 master_runner.cc:428] Detected that this master 580d354da5eb4d03a86d298eb6bc5af7 is joining an existing cluster
I20260810 22:34:11.953827 24567 master_runner.cc:432] Initiating AddMaster RPC to add 127.16.233.125:34439
I20260810 22:34:11.965320 24676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:11.956595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.125:34439 (local address 127.0.0.1:50912)
0810 22:34:11.956987 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:11.957002 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:11.957147 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:11.957441 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:11.957443 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:11.957673 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:11.957782 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:11.957787 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:11.958522 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:11.958529 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:11.959095 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:11.959102 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:11.959620 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:11.959623 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:11.959745 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:11.960773 (+ 1028us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:11.960792 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:11.961285 (+ 493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:34:11.963449 (+ 2164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:11.963452 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:11.963454 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:11.963707 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:11.964004 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:11.964007 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:11.964009 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:11.964105 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:11.964588 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:34:11.964593 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:34:11.964696 (+ 103us) client_negotiation.cc:769] Sending connection context
0810 22:34:11.964855 (+ 159us) client_negotiation.cc:240] Negotiation successful
0810 22:34:11.965025 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"mutex_wait_us":24,"thread_start_us":135,"threads_started":1}
I20260810 22:34:11.965299 24656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:11.957130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50912 (local address 127.16.233.125:34439)
0810 22:34:11.957249 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:11.957253 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:11.957272 (+ 19us) server_negotiation.cc:407] Connection header received
0810 22:34:11.957301 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:11.957303 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:11.957337 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:11.957426 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:11.957898 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:11.958412 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:11.959213 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:11.959488 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:11.959607 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:11.961483 (+ 1876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:11.961487 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:11.961490 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:11.961515 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:11.963335 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:11.963814 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:11.963818 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:11.963820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:11.963908 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:11.964201 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:11.964204 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:11.964206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:11.964482 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:11.964584 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:11.964919 (+ 335us) server_negotiation.cc:299] Negotiation successful
0810 22:34:11.965115 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":50}
I20260810 22:34:11.969739 24677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:11.956609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39177 (local address 127.0.0.1:45602)
0810 22:34:11.961031 (+ 4422us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:11.961046 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:11.961178 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:11.961597 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:11.961600 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:11.961805 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:11.961940 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:11.961951 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:11.962908 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:11.962910 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:11.963785 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:11.963804 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:11.964403 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:11.964407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:11.964499 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:11.965043 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:11.965055 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:11.965960 (+ 905us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:34:11.968024 (+ 2064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:11.968029 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:11.968031 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:11.968291 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:11.968596 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:11.968601 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:11.968603 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:11.968692 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:11.969140 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:34:11.969143 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:34:11.969271 (+ 128us) client_negotiation.cc:769] Sending connection context
0810 22:34:11.969390 (+ 119us) client_negotiation.cc:240] Negotiation successful
0810 22:34:11.969538 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4351,"spinlock_wait_cycles":9472,"thread_start_us":140,"threads_started":1}
I20260810 22:34:11.969775 24098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:11.957137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45602 (local address 127.16.233.126:39177)
0810 22:34:11.957265 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:11.957270 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:11.961144 (+ 3874us) server_negotiation.cc:407] Connection header received
0810 22:34:11.961309 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:11.961312 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:11.961392 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:11.961475 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:11.962099 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:11.962776 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:11.963917 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:11.964278 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:11.964404 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:11.966121 (+ 1717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:11.966125 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:11.966128 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:11.966160 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:11.967903 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:11.968406 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:11.968410 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:11.968412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:11.968501 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:11.968785 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:11.968789 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:11.968791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:11.969035 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:11.969153 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:11.969451 (+ 298us) server_negotiation.cc:299] Negotiation successful
0810 22:34:11.969571 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Aug 10 22:34:11 dist-test-slave-77v9 krb5kdc[24006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401250, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.126@KRBTEST.COM for oryx/127.16.233.125@KRBTEST.COM
WARNING: no policy specified for oryx/127.16.233.124@KRBTEST.COM; defaulting to no policy
I20260810 22:34:11.985834 24656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:11.975465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50914 (local address 127.16.233.125:34439)
0810 22:34:11.975583 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:11.975588 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:11.976618 (+ 1030us) server_negotiation.cc:407] Connection header received
0810 22:34:11.976889 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:11.976892 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:11.976951 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:11.977044 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:11.977881 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:11.978395 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:11.979307 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:11.979535 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:11.979643 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:11.982220 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:11.982223 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:11.982225 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:11.982250 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:11.983776 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:11.984345 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:11.984348 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:11.984350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:11.984434 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:11.984856 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:11.984859 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:11.984861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:11.985059 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:11.985173 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:11.985535 (+ 362us) server_negotiation.cc:299] Negotiation successful
0810 22:34:11.985658 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 22:34:11.986025 24678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:11.975455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.125:34439 (local address 127.0.0.1:50914)
0810 22:34:11.976466 (+ 1011us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:11.976531 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:11.976716 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:11.977085 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:11.977091 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:11.977600 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:11.977722 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:11.977729 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:11.978511 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:11.978516 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:11.979168 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:11.979178 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:11.979662 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:11.979666 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:11.979798 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:11.980447 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:11.980460 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:11.982052 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:34:11.983968 (+ 1916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:11.983978 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:11.983983 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:11.984228 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:11.984532 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:11.984536 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:11.984538 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:11.984635 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:11.985174 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:34:11.985185 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:34:11.985322 (+ 137us) client_negotiation.cc:769] Sending connection context
0810 22:34:11.985481 (+ 159us) client_negotiation.cc:240] Negotiation successful
0810 22:34:11.985703 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"thread_start_us":95,"threads_started":1}
I20260810 22:34:11.987784 24057 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.16.233.125:34439: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "580d354da5eb4d03a86d298eb6bc5af7"
member_type: NON_VOTER
last_known_addr {
host: "127.16.233.125"
port: 34439
}
attrs {
promote: true
}
}
dest_uuid: "61c3a2bed2604ebcb00d2c5647841a3c"
cas_config_opid_index: -1
I20260810 22:34:11.988638 24057 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61c3a2bed2604ebcb00d2c5647841a3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39177 } } peers { permanent_uuid: "580d354da5eb4d03a86d298eb6bc5af7" member_type: NON_VOTER last_known_addr { host: "127.16.233.125" port: 34439 } attrs { promote: true } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.233.124@KRBTEST.COM" created.
I20260810 22:34:11.990988 24682 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 61c3a2bed2604ebcb00d2c5647841a3c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 580d354da5eb4d03a86d298eb6bc5af7 (127.16.233.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39177 } } peers { permanent_uuid: "580d354da5eb4d03a86d298eb6bc5af7" member_type: NON_VOTER last_known_addr { host: "127.16.233.125" port: 34439 } attrs { promote: true } } }
I20260810 22:34:11.991755 24617 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:34:11.993206 24682 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.16.233.125:34439
I20260810 22:34:11.993500 24617 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7 [term 1 FOLLOWER]: Refusing update from remote peer 61c3a2bed2604ebcb00d2c5647841a3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260810 22:34:11.993723 24682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61c3a2bed2604ebcb00d2c5647841a3c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39177 } } peers { permanent_uuid: "580d354da5eb4d03a86d298eb6bc5af7" member_type: NON_VOTER last_known_addr { host: "127.16.233.125" port: 34439 } attrs { promote: true } } }
I20260810 22:34:11.993943 24682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61c3a2bed2604ebcb00d2c5647841a3c [sys.catalog]: This master's current role is: LEADER
I20260810 22:34:11.994189 24567 master.cc:537] Master@127.16.233.125:34439 shutting down...
I20260810 22:34:11.994920 24683 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 61c3a2bed2604ebcb00d2c5647841a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "580d354da5eb4d03a86d298eb6bc5af7" member_type: NON_VOTER last_known_addr { host: "127.16.233.125" port: 34439 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 22:34:11.997206 24040 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260810 22:34:11.998476 24040 consensus_peers.cc:487] T 00000000000000000000000000000000 P 61c3a2bed2604ebcb00d2c5647841a3c -> Peer 580d354da5eb4d03a86d298eb6bc5af7 (127.16.233.125:34439): Couldn't send request to peer 580d354da5eb4d03a86d298eb6bc5af7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260810 22:34:11.999221 24567 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:34:11.999527 24567 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:34:11.999658 24567 tablet_replica.cc:330] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.233.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/krb5kdc/oryx_127.16.233.124.keytab.
Entry for principal oryx/127.16.233.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/krb5kdc/oryx_127.16.233.124.keytab.
I20260810 22:34:12.012630 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.124:37989
--webserver_interface=127.16.233.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/krb5kdc/oryx_127.16.233.124.keytab
--principal=oryx/127.16.233.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:33445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.126:39177,127.16.233.125:34439,127.16.233.124:37989
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 22:34:12.014815 24567 master.cc:552] Master@127.16.233.125:34439 shutdown complete.
I20260810 22:34:12.014962 24567 master_runner.cc:305] Clearing existing system tablet
I20260810 22:34:12.015236 24567 fs_manager.cc:359] Metadata directory not provided
I20260810 22:34:12.015324 24567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:34:12.016541 24567 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 22:34:12.017578 24689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:12.017832 24567 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:34:12.017967 24567 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 22:34:12.018006 24567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/master-1/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/master-1/wal
uuid: "580d354da5eb4d03a86d298eb6bc5af7"
format_stamp: "Formatted at 2026-08-10 22:34:11 on dist-test-slave-77v9"
server_key: "12d392cdda1221b99b8ee637a03f163d"
server_key_iv: "b51bb565f9e3cf76bbe1022502bf26eb"
server_key_version: "encryptionkey@0"
I20260810 22:34:12.020047 24567 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:34:12.022403 24567 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260810 22:34:12.022531 24567 log.cc:1196] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7: Deleting WAL directory at /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260810 22:34:12.022790 24567 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7: Deleting consensus metadata
I20260810 22:34:12.022992 24567 master_runner.cc:315] Copying system tablet from 127.16.233.126:39177
I20260810 22:34:12.024427 24567 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7: tablet copy: Beginning tablet copy session from remote peer at address 127.16.233.126:39177
I20260810 22:34:12.033414 24695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:12.025149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39177 (local address 127.0.0.1:45612)
0810 22:34:12.025484 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:12.025500 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:12.025651 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:12.026256 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:12.026259 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:12.026481 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:12.026594 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:12.026600 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:12.027387 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:12.027392 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:12.027980 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:12.027988 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:12.028481 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:12.028485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:12.028684 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:12.029321 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:12.029333 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:12.029670 (+ 337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:34:12.031683 (+ 2013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:12.031686 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:12.031688 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:12.031944 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:12.032251 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:12.032255 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:12.032257 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:12.032349 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:12.032759 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:34:12.032762 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:34:12.032916 (+ 154us) client_negotiation.cc:769] Sending connection context
0810 22:34:12.033031 (+ 115us) client_negotiation.cc:240] Negotiation successful
0810 22:34:12.033204 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":94,"threads_started":1}
I20260810 22:34:12.033404 24098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:12.025235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45612 (local address 127.16.233.126:39177)
0810 22:34:12.025401 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:12.025406 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:12.025599 (+ 193us) server_negotiation.cc:407] Connection header received
0810 22:34:12.026064 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:12.026067 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:12.026125 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:12.026220 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:12.026726 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:12.027262 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:12.028104 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:12.028347 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:12.028466 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:12.029826 (+ 1360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:12.029830 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:12.029832 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:12.029864 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:12.031525 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:12.032051 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:12.032056 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:12.032058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:12.032150 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:12.032453 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:12.032456 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:12.032458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:12.032660 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:12.032772 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:12.033090 (+ 318us) server_negotiation.cc:299] Negotiation successful
0810 22:34:12.033211 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 22:34:12.034685 24078 tablet_copy_service.cc:140] P 61c3a2bed2604ebcb00d2c5647841a3c: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 580d354da5eb4d03a86d298eb6bc5af7 ({username='oryx', principal='oryx/127.16.233.125@KRBTEST.COM'} at 127.0.0.1:45612)
I20260810 22:34:12.034902 24078 tablet_copy_service.cc:161] P 61c3a2bed2604ebcb00d2c5647841a3c: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 580d354da5eb4d03a86d298eb6bc5af7 at {username='oryx', principal='oryx/127.16.233.125@KRBTEST.COM'} at 127.0.0.1:45612: session id = 580d354da5eb4d03a86d298eb6bc5af7-00000000000000000000000000000000
I20260810 22:34:12.036616 24078 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 61c3a2bed2604ebcb00d2c5647841a3c: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 22:34:12.038552 24567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260810 22:34:12.041374 24567 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7: tablet copy: Starting download of 0 data blocks...
I20260810 22:34:12.041631 24567 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7: tablet copy: Starting download of 1 WAL segments...
I20260810 22:34:12.043772 24567 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 22:34:12.045701 24078 tablet_copy_service.cc:342] P 61c3a2bed2604ebcb00d2c5647841a3c: Request end of tablet copy session 580d354da5eb4d03a86d298eb6bc5af7-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.16.233.125@KRBTEST.COM'} at 127.0.0.1:45612
I20260810 22:34:12.045856 24078 tablet_copy_service.cc:433] P 61c3a2bed2604ebcb00d2c5647841a3c: ending tablet copy session 580d354da5eb4d03a86d298eb6bc5af7-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 580d354da5eb4d03a86d298eb6bc5af7
I20260810 22:34:12.055119 24567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:34:12.057173 24696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:12.057554 24697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:12.058247 24699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:12.061020 24567 server_base.cc:1034] running on GCE node
I20260810 22:34:12.061219 24567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:34:12.061582 24567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:34:12.062822 24567 hybrid_clock.cc:648] HybridClock initialized: now 1786401252062788 us; error 44 us; skew 500 ppm
Aug 10 22:34:12 dist-test-slave-77v9 krb5kdc[24006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401252, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:34:12.064512 24567 init.cc:377] Logged in from keytab as oryx/127.16.233.125@KRBTEST.COM (short username oryx)
I20260810 22:34:12.065125 24567 webserver.cc:460] Webserver started at http://127.16.233.125:42639/ using document root <none> and password file <none>
I20260810 22:34:12.065387 24567 fs_manager.cc:359] Metadata directory not provided
I20260810 22:34:12.065435 24567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:34:12.067010 24567 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 22:34:12.068504 24710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:12.069073 24567 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 22:34:12.069159 24567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/master-1/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/master-1/wal
uuid: "580d354da5eb4d03a86d298eb6bc5af7"
format_stamp: "Formatted at 2026-08-10 22:34:11 on dist-test-slave-77v9"
server_key: "12d392cdda1221b99b8ee637a03f163d"
server_key_iv: "b51bb565f9e3cf76bbe1022502bf26eb"
server_key_version: "encryptionkey@0"
I20260810 22:34:12.069373 24567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:34:12.088563 24567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:34:12.091233 24567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:34:12.096478 24567 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.125:34439
I20260810 22:34:12.096900 24765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.125:34439 every 8 connection(s)
I20260810 22:34:12.097239 24567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/master-1/data/info.pb
I20260810 22:34:12.098755 24766 sys_catalog.cc:263] Verifying existing consensus state
I20260810 22:34:12.099630 24766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7: Bootstrap starting.
I20260810 22:34:12.109086 24766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 22:34:12.109481 24766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7: Bootstrap complete.
I20260810 22:34:12.110165 24766 raft_consensus.cc:348] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39177 } } peers { permanent_uuid: "580d354da5eb4d03a86d298eb6bc5af7" member_type: NON_VOTER last_known_addr { host: "127.16.233.125" port: 34439 } attrs { promote: true } }
I20260810 22:34:12.110312 24766 raft_consensus.cc:729] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 580d354da5eb4d03a86d298eb6bc5af7, State: Initialized, Role: LEARNER
I20260810 22:34:12.110440 24766 consensus_queue.cc:260] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39177 } } peers { permanent_uuid: "580d354da5eb4d03a86d298eb6bc5af7" member_type: NON_VOTER last_known_addr { host: "127.16.233.125" port: 34439 } attrs { promote: true } }
I20260810 22:34:12.110868 24769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39177 } } peers { permanent_uuid: "580d354da5eb4d03a86d298eb6bc5af7" member_type: NON_VOTER last_known_addr { host: "127.16.233.125" port: 34439 } attrs { promote: true } } }
I20260810 22:34:12.111047 24766 sys_catalog.cc:564] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:34:12.111064 24769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7 [sys.catalog]: This master's current role is: LEARNER
I20260810 22:34:12.114398 24780 catalog_manager.cc:1259] Loaded cluster ID: 0d14ee0609a54fd5bfab4629910fd236
I20260810 22:34:12.114812 24780 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7: loading cluster ID for follower catalog manager: success
I20260810 22:34:12.116575 24780 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7: acquiring CA information for follower catalog manager: success
I20260810 22:34:12.118096 24780 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 22:34:12.143585 24268 heartbeater.cc:503] Master 127.16.233.126:39177 was elected leader, sending a full tablet report...
W20260810 22:34:12.175550 24687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:34:12.175922 24687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:34:12.176017 24687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:34:12.180541 24687 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:34:12.180660 24687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:34:12.180701 24687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:34:12.180749 24687 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:34:12.180799 24687 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:34:12.186765 24687 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:33445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.126:39177,127.16.233.125:34439,127.16.233.124:37989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/krb5kdc/oryx_127.16.233.124.keytab
--principal=oryx/127.16.233.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.124:37989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.233.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:34:12.188126 24687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:34:12.189329 24687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:34:12.198378 24785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:12.198472 24786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:12.198395 24789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:12.198786 24687 server_base.cc:1034] running on GCE node
I20260810 22:34:12.199420 24687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:34:12.200592 24687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:34:12.201774 24687 hybrid_clock.cc:648] HybridClock initialized: now 1786401252201773 us; error 89 us; skew 500 ppm
Aug 10 22:34:12 dist-test-slave-77v9 krb5kdc[24006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401252, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:34:12.204792 24687 init.cc:377] Logged in from keytab as oryx/127.16.233.124@KRBTEST.COM (short username oryx)
I20260810 22:34:12.205950 24687 webserver.cc:460] Webserver started at http://127.16.233.124:46861/ using document root <none> and password file <none>
I20260810 22:34:12.206589 24687 fs_manager.cc:359] Metadata directory not provided
I20260810 22:34:12.206661 24687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:34:12.206898 24687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:34:12.209373 24687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/master-2/data/instance:
uuid: "8059d406ab3242d3868f653160912ca1"
format_stamp: "Formatted at 2026-08-10 22:34:12 on dist-test-slave-77v9"
server_key: "6042bc85b06468ea6eb6f3e14c141aa1"
server_key_iv: "36c2ed9da5da1e38abf8ebedb0d2c5c6"
server_key_version: "encryptionkey@0"
I20260810 22:34:12.210052 24687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/master-2/wal/instance:
uuid: "8059d406ab3242d3868f653160912ca1"
format_stamp: "Formatted at 2026-08-10 22:34:12 on dist-test-slave-77v9"
server_key: "6042bc85b06468ea6eb6f3e14c141aa1"
server_key_iv: "36c2ed9da5da1e38abf8ebedb0d2c5c6"
server_key_version: "encryptionkey@0"
I20260810 22:34:12.214639 24687 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 22:34:12.217500 24797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:12.219113 24687 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:34:12.219313 24687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/master-2/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/master-2/wal
uuid: "8059d406ab3242d3868f653160912ca1"
format_stamp: "Formatted at 2026-08-10 22:34:12 on dist-test-slave-77v9"
server_key: "6042bc85b06468ea6eb6f3e14c141aa1"
server_key_iv: "36c2ed9da5da1e38abf8ebedb0d2c5c6"
server_key_version: "encryptionkey@0"
I20260810 22:34:12.219472 24687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:34:12.240223 24687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:34:12.244489 24687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:34:12.244765 24687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:34:12.253335 24687 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.124:37989
I20260810 22:34:12.253656 24857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.124:37989 every 8 connection(s)
I20260810 22:34:12.254540 24687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/master-2/data/info.pb
I20260810 22:34:12.257803 24858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:34:12.262305 24858 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39177 } has no permanent_uuid. Determining permanent_uuid...
I20260810 22:34:12.262979 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 24687
Aug 10 22:34:12 dist-test-slave-77v9 krb5kdc[24006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401252, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.124@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM
Aug 10 22:34:12 dist-test-slave-77v9 krb5kdc[24006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.233.124@KRBTEST.COM
I20260810 22:34:12.278358 24098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:12.264748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45616 (local address 127.16.233.126:39177)
0810 22:34:12.264924 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:12.264929 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:12.265417 (+ 488us) server_negotiation.cc:407] Connection header received
0810 22:34:12.266301 (+ 884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:12.266304 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:12.266367 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:12.266472 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:12.267627 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:12.268506 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:12.270207 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:12.270544 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:12.270666 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:12.273920 (+ 3254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:12.273923 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:12.273925 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:12.273954 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:12.275557 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:12.276199 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:12.276203 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:12.276205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:12.276340 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:12.276948 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:12.276954 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:12.276956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:12.277190 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:12.277319 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:12.278008 (+ 689us) server_negotiation.cc:299] Negotiation successful
0810 22:34:12.278157 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260810 22:34:12.279311 24862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:12.264670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39177 (local address 127.0.0.1:45616)
0810 22:34:12.265251 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:12.265303 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:12.266075 (+ 772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:12.266556 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:12.266562 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:12.267093 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:12.267483 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:12.267496 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:12.268652 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:12.268655 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:12.270079 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:12.270088 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:12.270896 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:12.270904 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:12.271077 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:12.271580 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:12.271604 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:12.273728 (+ 2124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:34:12.275680 (+ 1952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:12.275690 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:12.275703 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:12.276028 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:12.276495 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:12.276498 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:12.276501 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:12.276743 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:12.277318 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:34:12.277327 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:34:12.277701 (+ 374us) client_negotiation.cc:769] Sending connection context
0810 22:34:12.277935 (+ 234us) client_negotiation.cc:240] Negotiation successful
0810 22:34:12.278328 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"spinlock_wait_cycles":37120,"thread_start_us":236,"threads_started":1}
I20260810 22:34:12.279978 24860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:12.264668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52354 (local address 127.16.233.124:37989)
0810 22:34:12.265168 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:12.265189 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:12.265222 (+ 33us) server_negotiation.cc:407] Connection header received
0810 22:34:12.266016 (+ 794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:12.266035 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:12.266257 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:12.266473 (+ 216us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:12.267464 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:12.268777 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:12.270012 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:12.270544 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:12.270713 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:12.274332 (+ 3619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:12.274342 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:12.274359 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:12.274391 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:12.276326 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:12.276875 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:12.276882 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:12.276883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:12.276967 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:12.277326 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:12.277331 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:12.277333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:12.277660 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:12.277873 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:12.278116 (+ 243us) server_negotiation.cc:299] Negotiation successful
0810 22:34:12.279776 (+ 1660us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1473,"server-negotiator.queue_time_us":359,"thread_start_us":202,"threads_started":1}
I20260810 22:34:12.281112 24858 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.233.125" port: 34439 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 22:34:12 dist-test-slave-77v9 krb5kdc[24006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401252, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.124@KRBTEST.COM for oryx/127.16.233.125@KRBTEST.COM
I20260810 22:34:12.291144 24862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:12.281667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.125:34439 (local address 127.0.0.1:50926)
0810 22:34:12.281799 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:12.281815 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:12.281902 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:12.282426 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:12.282429 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:12.282604 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:12.282716 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:12.282721 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:12.283673 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:12.283676 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:12.284278 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:12.284288 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:12.284962 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:12.284966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:12.285061 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:12.285585 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:12.285600 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:12.287151 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:34:12.289360 (+ 2209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:12.289364 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:12.289366 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:12.289591 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:12.289944 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:12.289946 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:12.289948 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:12.290044 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:12.290542 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:34:12.290546 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:34:12.290652 (+ 106us) client_negotiation.cc:769] Sending connection context
0810 22:34:12.290763 (+ 111us) client_negotiation.cc:240] Negotiation successful
0810 22:34:12.290930 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260810 22:34:12.291208 24863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:12.281833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50926 (local address 127.16.233.125:34439)
0810 22:34:12.282157 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:12.282161 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:12.282174 (+ 13us) server_negotiation.cc:407] Connection header received
0810 22:34:12.282210 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:12.282212 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:12.282280 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:12.282377 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:12.282837 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:12.283538 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:12.284428 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:12.284703 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:12.284890 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:12.287336 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:12.287339 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:12.287341 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:12.287367 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:12.289230 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:12.289702 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:12.289706 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:12.289708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:12.289826 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:12.290144 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:12.290146 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:12.290147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:12.290445 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:12.290551 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:12.290893 (+ 342us) server_negotiation.cc:299] Negotiation successful
0810 22:34:12.291055 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":136,"threads_started":1}
I20260810 22:34:12.292064 24858 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.233.124" port: 37989 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 22:34:12 dist-test-slave-77v9 krb5kdc[24006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401252, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.124@KRBTEST.COM for oryx/127.16.233.124@KRBTEST.COM
I20260810 22:34:12.300457 24860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:12.292683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52370 (local address 127.16.233.124:37989)
0810 22:34:12.292902 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:12.292907 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:12.292920 (+ 13us) server_negotiation.cc:407] Connection header received
0810 22:34:12.293023 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:12.293026 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:12.293068 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:12.293149 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:12.293549 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:12.294090 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:12.294826 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:12.295070 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:12.295183 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:12.297232 (+ 2049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:12.297236 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:12.297238 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:12.297259 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:12.298739 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:12.299186 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:12.299191 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:12.299193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:12.299268 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:12.299583 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:12.299586 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:12.299588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:12.299778 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:12.299891 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:12.300125 (+ 234us) server_negotiation.cc:299] Negotiation successful
0810 22:34:12.300293 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":69,"server-negotiator.queue_time_us":41}
I20260810 22:34:12.300457 24862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:12.292646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.124:37989 (local address 127.0.0.1:52370)
0810 22:34:12.292788 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:12.292802 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:12.292929 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:12.293165 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:12.293168 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:12.293362 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:12.293444 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:12.293449 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:12.294209 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:12.294213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:12.294712 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:12.294719 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:12.295182 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:12.295185 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:12.295251 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:12.295773 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:12.295793 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:12.297082 (+ 1289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:34:12.298885 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:12.298889 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:12.298891 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:12.299090 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:12.299395 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:12.299397 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:12.299399 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:12.299485 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:12.299878 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:34:12.299881 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:34:12.299962 (+ 81us) client_negotiation.cc:769] Sending connection context
0810 22:34:12.300079 (+ 117us) client_negotiation.cc:240] Negotiation successful
0810 22:34:12.300234 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260810 22:34:12.304642 24858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1: Bootstrap starting.
I20260810 22:34:12.307459 24858 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1: Neither blocks nor log segments found. Creating new log.
I20260810 22:34:12.308358 24858 log.cc:824] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1: Log is configured to *not* fsync() on all Append() calls
I20260810 22:34:12.311259 24858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1: No bootstrap required, opened a new log
I20260810 22:34:12.314895 24858 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39177 } } peers { permanent_uuid: "580d354da5eb4d03a86d298eb6bc5af7" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 34439 } } peers { permanent_uuid: "8059d406ab3242d3868f653160912ca1" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 37989 } }
I20260810 22:34:12.315151 24858 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:34:12.315188 24858 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8059d406ab3242d3868f653160912ca1, State: Initialized, Role: FOLLOWER
I20260810 22:34:12.315687 24858 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39177 } } peers { permanent_uuid: "580d354da5eb4d03a86d298eb6bc5af7" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 34439 } } peers { permanent_uuid: "8059d406ab3242d3868f653160912ca1" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 37989 } }
I20260810 22:34:12.316280 24865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39177 } } peers { permanent_uuid: "580d354da5eb4d03a86d298eb6bc5af7" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 34439 } } peers { permanent_uuid: "8059d406ab3242d3868f653160912ca1" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 37989 } } }
I20260810 22:34:12.316452 24865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 22:34:12.317153 24858 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1 [sys.catalog]: configured and running, proceeding with master startup.
W20260810 22:34:12.322494 24877 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 22:34:12.322606 24877 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 22:34:12.326403 24687 master_runner.cc:428] Detected that this master 8059d406ab3242d3868f653160912ca1 is joining an existing cluster
I20260810 22:34:12.326521 24687 master_runner.cc:432] Initiating AddMaster RPC to add 127.16.233.124:37989
I20260810 22:34:12.333911 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 24124
I20260810 22:34:12.339655 24863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:12.329587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50934 (local address 127.16.233.125:34439)
0810 22:34:12.329743 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:12.329749 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:12.329823 (+ 74us) server_negotiation.cc:407] Connection header received
0810 22:34:12.330012 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:12.330016 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:12.330080 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:12.330180 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:12.330732 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:12.331333 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:12.332364 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:12.332639 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:12.332767 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:12.334978 (+ 2211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:12.334985 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:12.334991 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:12.335024 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:12.337378 (+ 2354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:12.337982 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:12.337991 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:12.337994 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:12.338110 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:12.338485 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:12.338489 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:12.338491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:12.338732 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:12.338813 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:12.339276 (+ 463us) server_negotiation.cc:299] Negotiation successful
0810 22:34:12.339440 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 22:34:12.340386 24885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:12.329012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.125:34439 (local address 127.0.0.1:50934)
0810 22:34:12.329725 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:12.329742 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:12.329882 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:12.330198 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:12.330201 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:12.330463 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:12.330587 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:12.330593 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:12.331465 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:12.331472 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:12.332212 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:12.332222 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:12.332801 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:12.332834 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:12.332960 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:12.334251 (+ 1291us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:12.334280 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:12.334804 (+ 524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:34:12.337507 (+ 2703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:12.337511 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:12.337513 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:12.337827 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:12.338250 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:12.338257 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:12.338273 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:12.338376 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:12.338893 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:34:12.338896 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:34:12.338993 (+ 97us) client_negotiation.cc:769] Sending connection context
0810 22:34:12.339987 (+ 994us) client_negotiation.cc:240] Negotiation successful
0810 22:34:12.340165 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":141,"threads_started":1}
I20260810 22:34:12.348866 24860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:12.329180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52384 (local address 127.16.233.124:37989)
0810 22:34:12.329292 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:12.329296 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:12.337260 (+ 7964us) server_negotiation.cc:407] Connection header received
0810 22:34:12.337447 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:12.337454 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:12.337505 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:12.337559 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:12.338320 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:12.339127 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:12.341071 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:12.341403 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:12.341481 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:12.343230 (+ 1749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:12.343234 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:12.343237 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:12.343267 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:12.345480 (+ 2213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:12.346514 (+ 1034us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:12.346518 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:12.346520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:12.346624 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:12.346958 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:12.346961 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:12.346964 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:12.347585 (+ 621us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:12.347674 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:12.348443 (+ 769us) server_negotiation.cc:299] Negotiation successful
0810 22:34:12.348582 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33,"spinlock_wait_cycles":896}
I20260810 22:34:12.349756 24884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:12.329017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.124:37989 (local address 127.0.0.1:52384)
0810 22:34:12.337150 (+ 8133us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:12.337171 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:12.337336 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:12.337754 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:12.337757 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:12.338016 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:12.338193 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:12.338205 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:12.339256 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:12.339259 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:12.340933 (+ 1674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:12.340947 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:12.341630 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:12.341633 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:12.341745 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:12.342414 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:12.342430 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:12.342944 (+ 514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:34:12.345965 (+ 3021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:12.345968 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:12.345971 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:12.346368 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:12.346741 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:12.346744 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:12.346746 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:12.346850 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:12.348084 (+ 1234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:34:12.348090 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:34:12.348236 (+ 146us) client_negotiation.cc:769] Sending connection context
0810 22:34:12.349360 (+ 1124us) client_negotiation.cc:240] Negotiation successful
0810 22:34:12.349522 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8021,"mutex_wait_us":25,"thread_start_us":115,"threads_started":1}
I20260810 22:34:12.350984 24885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:12.329407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39177 (local address 127.0.0.1:45624)
0810 22:34:12.341001 (+ 11594us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:12.341021 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:12.341172 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:12.341614 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:12.341617 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:12.341847 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:12.341992 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:12.342005 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:12.343283 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:12.343287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:12.344318 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:12.344335 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:12.345297 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:12.345300 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:12.345441 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:12.345979 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:12.345993 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:12.346552 (+ 559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:34:12.348994 (+ 2442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:12.348997 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:12.348999 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:12.349282 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:12.349672 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:12.349675 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:12.349677 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:12.349767 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:12.350456 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:34:12.350459 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:34:12.350538 (+ 79us) client_negotiation.cc:769] Sending connection context
0810 22:34:12.350671 (+ 133us) client_negotiation.cc:240] Negotiation successful
0810 22:34:12.350821 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11517,"thread_start_us":104,"threads_started":1}
I20260810 22:34:12.351155 24098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:12.329037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45624 (local address 127.16.233.126:39177)
0810 22:34:12.329318 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:12.329323 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:12.341092 (+ 11769us) server_negotiation.cc:407] Connection header received
0810 22:34:12.341307 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:12.341311 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:12.341375 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:12.341453 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:12.342162 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:12.343120 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:12.344494 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:12.344971 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:12.345085 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:12.346749 (+ 1664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:12.346755 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:12.346758 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:12.346818 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:12.348855 (+ 2037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:12.349423 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:12.349427 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:12.349429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:12.349542 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:12.349902 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:12.349908 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:12.349911 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:12.350161 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:12.350248 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:12.350735 (+ 487us) server_negotiation.cc:299] Negotiation successful
0810 22:34:12.350989 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199}
I20260810 22:34:12.365659 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.65:39951
--local_ip_for_outbound_sockets=127.16.233.65
--tserver_master_addrs=127.16.233.126:39177
--webserver_port=46001
--webserver_interface=127.16.233.65
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/krb5kdc/oryx_127.16.233.65.keytab
--principal=oryx/127.16.233.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:33445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.16.233.126:39177,127.16.233.125:34439
--tserver_master_addrs=127.16.233.126:39177,127.16.233.125:34439,127.16.233.124:37989 with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 22:34:12 dist-test-slave-77v9 krb5kdc[24006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401250, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.126@KRBTEST.COM for oryx/127.16.233.124@KRBTEST.COM
I20260810 22:34:12.389757 24678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:12.360482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.124:37989 (local address 127.0.0.1:52392)
0810 22:34:12.360777 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:12.360794 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:12.360919 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:12.361251 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:12.361257 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:12.361522 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:12.361668 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:12.361679 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:12.362740 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:12.362747 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:12.363843 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:12.363855 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:12.364662 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:12.364669 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:12.364860 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:12.365400 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:12.365420 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:12.367203 (+ 1783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:34:12.387543 (+ 20340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:12.387547 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:12.387549 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:12.387877 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:12.388461 (+ 584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:12.388464 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:12.388466 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:12.388572 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:12.389181 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:34:12.389194 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:34:12.389310 (+ 116us) client_negotiation.cc:769] Sending connection context
0810 22:34:12.389416 (+ 106us) client_negotiation.cc:240] Negotiation successful
0810 22:34:12.389586 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193}
I20260810 22:34:12.390226 24414 heartbeater.cc:503] Master 127.16.233.126:39177 was elected leader, sending a full tablet report...
I20260810 22:34:12.390777 24860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:12.360749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52392 (local address 127.16.233.124:37989)
0810 22:34:12.360906 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:12.360911 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:12.360926 (+ 15us) server_negotiation.cc:407] Connection header received
0810 22:34:12.361060 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:12.361063 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:12.361118 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:12.361204 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:12.361798 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:12.362617 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:12.363993 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:12.364421 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:12.364521 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:12.367394 (+ 2873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:12.367401 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:12.367405 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:12.367440 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:12.387299 (+ 19859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:12.388216 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:12.388220 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:12.388222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:12.388327 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:12.388691 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:12.388696 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:12.388700 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:12.389015 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:12.390267 (+ 1252us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:12.390369 (+ 102us) server_negotiation.cc:299] Negotiation successful
0810 22:34:12.390524 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260810 22:34:12.391993 24057 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.16.233.124:37989: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "8059d406ab3242d3868f653160912ca1"
member_type: NON_VOTER
last_known_addr {
host: "127.16.233.124"
port: 37989
}
attrs {
promote: true
}
}
dest_uuid: "61c3a2bed2604ebcb00d2c5647841a3c"
cas_config_opid_index: 5
I20260810 22:34:12.392438 24057 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61c3a2bed2604ebcb00d2c5647841a3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39177 } } peers { permanent_uuid: "580d354da5eb4d03a86d298eb6bc5af7" member_type: NON_VOTER last_known_addr { host: "127.16.233.125" port: 34439 } attrs { promote: true } } peers { permanent_uuid: "8059d406ab3242d3868f653160912ca1" member_type: NON_VOTER last_known_addr { host: "127.16.233.124" port: 37989 } attrs { promote: true } }
I20260810 22:34:12.393281 24863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:12.388032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50944 (local address 127.16.233.125:34439)
0810 22:34:12.388185 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:12.388189 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:12.388311 (+ 122us) server_negotiation.cc:407] Connection header received
0810 22:34:12.388509 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:12.388513 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:12.388577 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:12.388653 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:34:12.389207 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:12.390064 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:12.391640 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:12.392356 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:12.392458 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:12.392614 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:12.393058 (+ 444us) server_negotiation.cc:299] Negotiation successful
0810 22:34:12.393149 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 22:34:12.393932 24682 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 61c3a2bed2604ebcb00d2c5647841a3c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 8059d406ab3242d3868f653160912ca1 (127.16.233.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39177 } } peers { permanent_uuid: "580d354da5eb4d03a86d298eb6bc5af7" member_type: NON_VOTER last_known_addr { host: "127.16.233.125" port: 34439 } attrs { promote: true } } peers { permanent_uuid: "8059d406ab3242d3868f653160912ca1" member_type: NON_VOTER last_known_addr { host: "127.16.233.124" port: 37989 } attrs { promote: true } } }
I20260810 22:34:12.394294 24826 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:34:12.395653 24682 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.16.233.124:37989
I20260810 22:34:12.395880 24682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61c3a2bed2604ebcb00d2c5647841a3c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39177 } } peers { permanent_uuid: "580d354da5eb4d03a86d298eb6bc5af7" member_type: NON_VOTER last_known_addr { host: "127.16.233.125" port: 34439 } attrs { promote: true } } peers { permanent_uuid: "8059d406ab3242d3868f653160912ca1" member_type: NON_VOTER last_known_addr { host: "127.16.233.124" port: 37989 } attrs { promote: true } } }
I20260810 22:34:12.395998 24682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61c3a2bed2604ebcb00d2c5647841a3c [sys.catalog]: This master's current role is: LEADER
I20260810 22:34:12.396092 24826 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1 [term 1 FOLLOWER]: Refusing update from remote peer 61c3a2bed2604ebcb00d2c5647841a3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260810 22:34:12.396538 24887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:12.387616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.125:34439 (local address 127.0.0.1:50944)
0810 22:34:12.388222 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:12.388237 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:12.388387 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:12.388868 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:12.388871 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:12.388890 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:34:12.389071 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:12.389080 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:12.390215 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:12.390218 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:12.391483 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:12.391495 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:12.392721 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:12.392725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:12.392864 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:12.392927 (+ 63us) client_negotiation.cc:769] Sending connection context
0810 22:34:12.396267 (+ 3340us) client_negotiation.cc:240] Negotiation successful
0810 22:34:12.396388 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":107,"threads_started":1}
I20260810 22:34:12.397473 24739 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260810 22:34:12.397794 24683 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 61c3a2bed2604ebcb00d2c5647841a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8059d406ab3242d3868f653160912ca1" member_type: NON_VOTER last_known_addr { host: "127.16.233.124" port: 37989 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:34:12.398147 24738 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7 [term 1 LEARNER]: Refusing update from remote peer 61c3a2bed2604ebcb00d2c5647841a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260810 22:34:12.398382 24769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61c3a2bed2604ebcb00d2c5647841a3c. Latest consensus state: current_term: 1 leader_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39177 } } peers { permanent_uuid: "580d354da5eb4d03a86d298eb6bc5af7" member_type: NON_VOTER last_known_addr { host: "127.16.233.125" port: 34439 } attrs { promote: true } } }
I20260810 22:34:12.398499 24769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7 [sys.catalog]: This master's current role is: LEARNER
I20260810 22:34:12.399148 24687 master.cc:537] Master@127.16.233.124:37989 shutting down...
I20260810 22:34:12.399246 24683 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 61c3a2bed2604ebcb00d2c5647841a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "580d354da5eb4d03a86d298eb6bc5af7" member_type: NON_VOTER last_known_addr { host: "127.16.233.125" port: 34439 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260810 22:34:12.401352 24738 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 8059d406ab3242d3868f653160912ca1 (127.16.233.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39177 } } peers { permanent_uuid: "580d354da5eb4d03a86d298eb6bc5af7" member_type: NON_VOTER last_known_addr { host: "127.16.233.125" port: 34439 } attrs { promote: true } } peers { permanent_uuid: "8059d406ab3242d3868f653160912ca1" member_type: NON_VOTER last_known_addr { host: "127.16.233.124" port: 37989 } attrs { promote: true } } }
I20260810 22:34:12.403606 24826 raft_consensus.cc:858] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39177 } } peers { permanent_uuid: "580d354da5eb4d03a86d298eb6bc5af7" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 34439 } } peers { permanent_uuid: "8059d406ab3242d3868f653160912ca1" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 37989 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39177 } } peers { permanent_uuid: "580d354da5eb4d03a86d298eb6bc5af7" member_type: NON_VOTER last_known_addr { host: "127.16.233.125" port: 34439 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39177 } } peers { permanent_uuid: "580d354da5eb4d03a86d298eb6bc5af7" member_type: NON_VOTER last_known_addr { host: "127.16.233.125" port: 34439 } attrs { promote: true } } peers { permanent_uuid: "8059d406ab3242d3868f653160912ca1" member_type: NON_VOTER last_known_addr { host: "127.16.233.124" port: 37989 } attrs { promote: true } }
W20260810 22:34:12.403754 24826 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39177 } } peers { permanent_uuid: "580d354da5eb4d03a86d298eb6bc5af7" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 34439 } } peers { permanent_uuid: "8059d406ab3242d3868f653160912ca1" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 37989 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39177 } } peers { permanent_uuid: "580d354da5eb4d03a86d298eb6bc5af7" member_type: NON_VOTER last_known_addr { host: "127.16.233.125" port: 34439 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39177 } } peers { permanent_uuid: "580d354da5eb4d03a86d298eb6bc5af7" member_type: NON_VOTER last_known_addr { host: "127.16.233.125" port: 34439 } attrs { promote: true } } peers { permanent_uuid: "8059d406ab3242d3868f653160912ca1" member_type: NON_VOTER last_known_addr { host: "127.16.233.124" port: 37989 } attrs { promote: true } }
I20260810 22:34:12.405508 24769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39177 } } peers { permanent_uuid: "580d354da5eb4d03a86d298eb6bc5af7" member_type: NON_VOTER last_known_addr { host: "127.16.233.125" port: 34439 } attrs { promote: true } } peers { permanent_uuid: "8059d406ab3242d3868f653160912ca1" member_type: NON_VOTER last_known_addr { host: "127.16.233.124" port: 37989 } attrs { promote: true } } }
I20260810 22:34:12.405655 24769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7 [sys.catalog]: This master's current role is: LEARNER
I20260810 22:34:12.405475 24826 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 580d354da5eb4d03a86d298eb6bc5af7 (127.16.233.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39177 } } peers { permanent_uuid: "580d354da5eb4d03a86d298eb6bc5af7" member_type: NON_VOTER last_known_addr { host: "127.16.233.125" port: 34439 } attrs { promote: true } } }
I20260810 22:34:12.407500 24892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39177 } } peers { permanent_uuid: "580d354da5eb4d03a86d298eb6bc5af7" member_type: NON_VOTER last_known_addr { host: "127.16.233.125" port: 34439 } attrs { promote: true } } }
I20260810 22:34:12.407640 24892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260810 22:34:12.407732 24892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39177 } } peers { permanent_uuid: "580d354da5eb4d03a86d298eb6bc5af7" member_type: NON_VOTER last_known_addr { host: "127.16.233.125" port: 34439 } attrs { promote: true } } }
I20260810 22:34:12.407810 24892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260810 22:34:12.407953 24865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61c3a2bed2604ebcb00d2c5647841a3c. Latest consensus state: current_term: 1 leader_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39177 } } peers { permanent_uuid: "580d354da5eb4d03a86d298eb6bc5af7" member_type: NON_VOTER last_known_addr { host: "127.16.233.125" port: 34439 } attrs { promote: true } } }
I20260810 22:34:12.408113 24865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260810 22:34:12.411041 24683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61c3a2bed2604ebcb00d2c5647841a3c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39177 } } peers { permanent_uuid: "580d354da5eb4d03a86d298eb6bc5af7" member_type: NON_VOTER last_known_addr { host: "127.16.233.125" port: 34439 } attrs { promote: true } } peers { permanent_uuid: "8059d406ab3242d3868f653160912ca1" member_type: NON_VOTER last_known_addr { host: "127.16.233.124" port: 37989 } attrs { promote: true } } }
I20260810 22:34:12.411226 24683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61c3a2bed2604ebcb00d2c5647841a3c [sys.catalog]: This master's current role is: LEADER
I20260810 22:34:12.412729 24889 mvcc.cc:205] Tried to move back new op lower bound from 7317099523458981888 to 7317099523318067200. Current Snapshot: MvccSnapshot[applied={T|T < 7317099523356930048}]
W20260810 22:34:12.412941 24041 consensus_peers.cc:487] T 00000000000000000000000000000000 P 61c3a2bed2604ebcb00d2c5647841a3c -> Peer 8059d406ab3242d3868f653160912ca1 (127.16.233.124:37989): Couldn't send request to peer 8059d406ab3242d3868f653160912ca1. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260810 22:34:12.414278 24687 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260810 22:34:12.414314 24682 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 61c3a2bed2604ebcb00d2c5647841a3c: attempting to promote NON_VOTER 580d354da5eb4d03a86d298eb6bc5af7 to VOTER
I20260810 22:34:12.414544 24687 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260810 22:34:12.414644 24687 tablet_replica.cc:330] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1: stopping tablet replica
I20260810 22:34:12.414839 24682 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61c3a2bed2604ebcb00d2c5647841a3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39177 } } peers { permanent_uuid: "580d354da5eb4d03a86d298eb6bc5af7" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 34439 } attrs { promote: false } } peers { permanent_uuid: "8059d406ab3242d3868f653160912ca1" member_type: NON_VOTER last_known_addr { host: "127.16.233.124" port: 37989 } attrs { promote: true } }
I20260810 22:34:12.417366 24738 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7 [term 1 LEARNER]: Refusing update from remote peer 61c3a2bed2604ebcb00d2c5647841a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20260810 22:34:12.418326 24041 consensus_peers.cc:487] T 00000000000000000000000000000000 P 61c3a2bed2604ebcb00d2c5647841a3c -> Peer 8059d406ab3242d3868f653160912ca1 (127.16.233.124:37989): Couldn't send request to peer 8059d406ab3242d3868f653160912ca1. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260810 22:34:12.418597 24683 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 61c3a2bed2604ebcb00d2c5647841a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "580d354da5eb4d03a86d298eb6bc5af7" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 34439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260810 22:34:12.420789 24683 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 61c3a2bed2604ebcb00d2c5647841a3c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 580d354da5eb4d03a86d298eb6bc5af7 (127.16.233.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39177 } } peers { permanent_uuid: "580d354da5eb4d03a86d298eb6bc5af7" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 34439 } attrs { promote: false } } peers { permanent_uuid: "8059d406ab3242d3868f653160912ca1" member_type: NON_VOTER last_known_addr { host: "127.16.233.124" port: 37989 } attrs { promote: true } } }
I20260810 22:34:12.422746 24683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61c3a2bed2604ebcb00d2c5647841a3c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39177 } } peers { permanent_uuid: "580d354da5eb4d03a86d298eb6bc5af7" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 34439 } attrs { promote: false } } peers { permanent_uuid: "8059d406ab3242d3868f653160912ca1" member_type: NON_VOTER last_known_addr { host: "127.16.233.124" port: 37989 } attrs { promote: true } } }
I20260810 22:34:12.422888 24683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61c3a2bed2604ebcb00d2c5647841a3c [sys.catalog]: This master's current role is: LEADER
I20260810 22:34:12.423827 24682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61c3a2bed2604ebcb00d2c5647841a3c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39177 } } peers { permanent_uuid: "580d354da5eb4d03a86d298eb6bc5af7" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 34439 } attrs { promote: false } } peers { permanent_uuid: "8059d406ab3242d3868f653160912ca1" member_type: NON_VOTER last_known_addr { host: "127.16.233.124" port: 37989 } attrs { promote: true } } }
I20260810 22:34:12.423959 24682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61c3a2bed2604ebcb00d2c5647841a3c [sys.catalog]: This master's current role is: LEADER
I20260810 22:34:12.424958 24738 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 580d354da5eb4d03a86d298eb6bc5af7 (127.16.233.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39177 } } peers { permanent_uuid: "580d354da5eb4d03a86d298eb6bc5af7" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 34439 } attrs { promote: false } } peers { permanent_uuid: "8059d406ab3242d3868f653160912ca1" member_type: NON_VOTER last_known_addr { host: "127.16.233.124" port: 37989 } attrs { promote: true } } }
I20260810 22:34:12.426647 24769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39177 } } peers { permanent_uuid: "580d354da5eb4d03a86d298eb6bc5af7" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 34439 } attrs { promote: false } } peers { permanent_uuid: "8059d406ab3242d3868f653160912ca1" member_type: NON_VOTER last_known_addr { host: "127.16.233.124" port: 37989 } attrs { promote: true } } }
I20260810 22:34:12.426781 24769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 22:34:12.435079 24687 master.cc:552] Master@127.16.233.124:37989 shutdown complete.
I20260810 22:34:12.435154 24687 master_runner.cc:305] Clearing existing system tablet
I20260810 22:34:12.435410 24687 fs_manager.cc:359] Metadata directory not provided
I20260810 22:34:12.435472 24687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:34:12.436939 24687 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 22:34:12.437975 24900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:12.438287 24687 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:34:12.438423 24687 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 22:34:12.438458 24687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/master-2/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/master-2/wal
uuid: "8059d406ab3242d3868f653160912ca1"
format_stamp: "Formatted at 2026-08-10 22:34:12 on dist-test-slave-77v9"
server_key: "6042bc85b06468ea6eb6f3e14c141aa1"
server_key_iv: "36c2ed9da5da1e38abf8ebedb0d2c5c6"
server_key_version: "encryptionkey@0"
I20260810 22:34:12.440411 24687 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:34:12.443019 24687 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260810 22:34:12.443137 24687 log.cc:1196] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1: Deleting WAL directory at /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260810 22:34:12.443454 24687 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1: Deleting consensus metadata
I20260810 22:34:12.443634 24687 master_runner.cc:315] Copying system tablet from 127.16.233.126:39177
I20260810 22:34:12.444981 24687 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1: tablet copy: Beginning tablet copy session from remote peer at address 127.16.233.126:39177
I20260810 22:34:12.458127 24906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:12.445599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39177 (local address 127.0.0.1:45640)
0810 22:34:12.449060 (+ 3461us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:12.449080 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:12.449238 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:12.449564 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:12.449567 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:12.449818 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:12.449945 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:12.449952 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:12.451045 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:12.451049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:12.451922 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:12.451933 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:12.452576 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:12.452580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:12.452696 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:12.453357 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:12.453372 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:12.453890 (+ 518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:34:12.456274 (+ 2384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:12.456278 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:12.456281 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:12.456560 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:12.456926 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:12.456929 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:12.456931 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:12.457035 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:12.457513 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:34:12.457516 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:34:12.457634 (+ 118us) client_negotiation.cc:769] Sending connection context
0810 22:34:12.457750 (+ 116us) client_negotiation.cc:240] Negotiation successful
0810 22:34:12.457921 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3364,"thread_start_us":94,"threads_started":1}
I20260810 22:34:12.458165 24098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:12.445730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45640 (local address 127.16.233.126:39177)
0810 22:34:12.446031 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:12.446036 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:12.449210 (+ 3174us) server_negotiation.cc:407] Connection header received
0810 22:34:12.449351 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:12.449355 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:12.449419 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:12.449522 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:12.450097 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:12.450907 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:12.452075 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:12.452430 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:12.452553 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:12.454076 (+ 1523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:12.454081 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:12.454084 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:12.454113 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:12.456146 (+ 2033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:12.456680 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:12.456684 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:12.456689 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:12.456803 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:12.457143 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:12.457147 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:12.457149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:12.457399 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:12.457506 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:12.457843 (+ 337us) server_negotiation.cc:299] Negotiation successful
0810 22:34:12.457985 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195}
I20260810 22:34:12.459203 24078 tablet_copy_service.cc:140] P 61c3a2bed2604ebcb00d2c5647841a3c: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 8059d406ab3242d3868f653160912ca1 ({username='oryx', principal='oryx/127.16.233.124@KRBTEST.COM'} at 127.0.0.1:45640)
I20260810 22:34:12.459331 24078 tablet_copy_service.cc:161] P 61c3a2bed2604ebcb00d2c5647841a3c: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 8059d406ab3242d3868f653160912ca1 at {username='oryx', principal='oryx/127.16.233.124@KRBTEST.COM'} at 127.0.0.1:45640: session id = 8059d406ab3242d3868f653160912ca1-00000000000000000000000000000000
I20260810 22:34:12.460510 24078 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 61c3a2bed2604ebcb00d2c5647841a3c: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 22:34:12.462098 24687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260810 22:34:12.465129 24687 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1: tablet copy: Starting download of 0 data blocks...
I20260810 22:34:12.465512 24687 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1: tablet copy: Starting download of 1 WAL segments...
I20260810 22:34:12.467648 24687 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 22:34:12.470290 24078 tablet_copy_service.cc:342] P 61c3a2bed2604ebcb00d2c5647841a3c: Request end of tablet copy session 8059d406ab3242d3868f653160912ca1-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.16.233.124@KRBTEST.COM'} at 127.0.0.1:45640
I20260810 22:34:12.470414 24078 tablet_copy_service.cc:433] P 61c3a2bed2604ebcb00d2c5647841a3c: ending tablet copy session 8059d406ab3242d3868f653160912ca1-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 8059d406ab3242d3868f653160912ca1
I20260810 22:34:12.479431 24687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:34:12.481853 24907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:12.481897 24908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:12.482172 24687 server_base.cc:1034] running on GCE node
W20260810 22:34:12.482317 24910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:12.482630 24687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:34:12.483021 24687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:34:12.484184 24687 hybrid_clock.cc:648] HybridClock initialized: now 1786401252484171 us; error 34 us; skew 500 ppm
Aug 10 22:34:12 dist-test-slave-77v9 krb5kdc[24006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401252, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:34:12.486254 24687 init.cc:377] Logged in from keytab as oryx/127.16.233.124@KRBTEST.COM (short username oryx)
I20260810 22:34:12.487107 24687 webserver.cc:460] Webserver started at http://127.16.233.124:38195/ using document root <none> and password file <none>
I20260810 22:34:12.487421 24687 fs_manager.cc:359] Metadata directory not provided
I20260810 22:34:12.487533 24687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:34:12.489316 24687 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260810 22:34:12.490599 24918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:12.490979 24687 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 22:34:12.491119 24687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/master-2/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/master-2/wal
uuid: "8059d406ab3242d3868f653160912ca1"
format_stamp: "Formatted at 2026-08-10 22:34:12 on dist-test-slave-77v9"
server_key: "6042bc85b06468ea6eb6f3e14c141aa1"
server_key_iv: "36c2ed9da5da1e38abf8ebedb0d2c5c6"
server_key_version: "encryptionkey@0"
I20260810 22:34:12.491299 24687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:34:12.506788 24687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:34:12.509414 24687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:34:12.515362 24687 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.124:37989
I20260810 22:34:12.516258 24687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/master-2/data/info.pb
I20260810 22:34:12.516574 24979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.124:37989 every 8 connection(s)
I20260810 22:34:12.518066 24980 sys_catalog.cc:263] Verifying existing consensus state
I20260810 22:34:12.518982 24980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1: Bootstrap starting.
I20260810 22:34:12.526595 24980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 22:34:12.526924 24980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1: Bootstrap complete.
I20260810 22:34:12.527529 24980 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39177 } } peers { permanent_uuid: "580d354da5eb4d03a86d298eb6bc5af7" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 34439 } attrs { promote: false } } peers { permanent_uuid: "8059d406ab3242d3868f653160912ca1" member_type: NON_VOTER last_known_addr { host: "127.16.233.124" port: 37989 } attrs { promote: true } }
I20260810 22:34:12.527647 24980 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8059d406ab3242d3868f653160912ca1, State: Initialized, Role: LEARNER
I20260810 22:34:12.527766 24980 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39177 } } peers { permanent_uuid: "580d354da5eb4d03a86d298eb6bc5af7" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 34439 } attrs { promote: false } } peers { permanent_uuid: "8059d406ab3242d3868f653160912ca1" member_type: NON_VOTER last_known_addr { host: "127.16.233.124" port: 37989 } attrs { promote: true } }
I20260810 22:34:12.528223 24986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39177 } } peers { permanent_uuid: "580d354da5eb4d03a86d298eb6bc5af7" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 34439 } attrs { promote: false } } peers { permanent_uuid: "8059d406ab3242d3868f653160912ca1" member_type: NON_VOTER last_known_addr { host: "127.16.233.124" port: 37989 } attrs { promote: true } } }
I20260810 22:34:12.528326 24980 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:34:12.528399 24986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1 [sys.catalog]: This master's current role is: LEARNER
I20260810 22:34:12.531677 24998 catalog_manager.cc:1259] Loaded cluster ID: 0d14ee0609a54fd5bfab4629910fd236
I20260810 22:34:12.531760 24998 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1: loading cluster ID for follower catalog manager: success
W20260810 22:34:12.532804 24886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:34:12.533205 24886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:34:12.533347 24886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260810 22:34:12.534242 24998 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1: acquiring CA information for follower catalog manager: success
I20260810 22:34:12.535761 24998 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260810 22:34:12.539055 24886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:34:12.539177 24886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:34:12.539314 24886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.65
I20260810 22:34:12.545951 24886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:33445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/krb5kdc/oryx_127.16.233.65.keytab
--principal=oryx/127.16.233.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.65:39951
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.233.65
--webserver_port=46001
--tserver_master_addrs=127.16.233.126:39177,127.16.233.125:34439,127.16.233.124:37989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.65
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:34:12.547230 24886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:34:12.548328 24886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:34:12.559033 25002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:12.559073 24886 server_base.cc:1034] running on GCE node
W20260810 22:34:12.559026 25004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:12.559026 25001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:12.559787 24886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:34:12.560529 24886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:34:12.561759 24886 hybrid_clock.cc:648] HybridClock initialized: now 1786401252561700 us; error 70 us; skew 500 ppm
Aug 10 22:34:12 dist-test-slave-77v9 krb5kdc[24006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401252, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:34:12.565117 24886 init.cc:377] Logged in from keytab as oryx/127.16.233.65@KRBTEST.COM (short username oryx)
I20260810 22:34:12.566198 24886 webserver.cc:460] Webserver started at http://127.16.233.65:46001/ using document root <none> and password file <none>
I20260810 22:34:12.566917 24886 fs_manager.cc:359] Metadata directory not provided
I20260810 22:34:12.567046 24886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:34:12.571646 24886 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260810 22:34:12.574447 25012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:12.575975 24886 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 22:34:12.576154 24886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-0/wal
uuid: "599ede616cdd412fa79af64abd77799a"
format_stamp: "Formatted at 2026-08-10 22:34:11 on dist-test-slave-77v9"
server_key: "5da5dac1fd1333489c640283d02ce6e4"
server_key_iv: "5a93e3d275647d85d6f4df79b70e202e"
server_key_version: "encryptionkey@0"
I20260810 22:34:12.576905 24886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:34:12.599746 24886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:34:12.603978 24886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:34:12.604216 24886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:34:12.604930 24886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:34:12.605873 24886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:34:12.605923 24886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:12.605964 24886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:34:12.605980 24886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:12.616394 24886 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.65:39951
I20260810 22:34:12.616514 25133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.65:39951 every 8 connection(s)
I20260810 22:34:12.617704 24886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
I20260810 22:34:12.627815 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 24886
I20260810 22:34:12.628039 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 24273
Aug 10 22:34:12 dist-test-slave-77v9 krb5kdc[24006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401252, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.65@KRBTEST.COM for oryx/127.16.233.124@KRBTEST.COM
Aug 10 22:34:12 dist-test-slave-77v9 krb5kdc[24006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401252, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.65@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM
I20260810 22:34:12.637333 25140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:12.621193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:35159 (local address 127.16.233.124:37989)
0810 22:34:12.621498 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:12.621502 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:12.622350 (+ 848us) server_negotiation.cc:407] Connection header received
0810 22:34:12.623317 (+ 967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:12.623320 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:12.623382 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:12.623508 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:12.625232 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:12.625993 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:12.627504 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:12.627786 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:12.627900 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:12.631625 (+ 3725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:12.631629 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:12.631632 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:12.631663 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:12.633719 (+ 2056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:12.634438 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:12.634442 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:12.634444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:12.634542 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:12.634977 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:12.634980 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:12.634982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:12.635238 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:12.635337 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:12.637003 (+ 1666us) server_negotiation.cc:299] Negotiation successful
0810 22:34:12.637151 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":115,"threads_started":1}
I20260810 22:34:12.638082 25142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:12.620640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.124:37989 (local address 127.16.233.65:35159)
0810 22:34:12.622182 (+ 1542us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:12.622230 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:12.623120 (+ 890us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:12.623705 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:12.623715 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:12.624342 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:12.625072 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:12.625092 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:12.626166 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:12.626169 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:12.627362 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:12.627372 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:12.628072 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:12.628098 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:12.628399 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:12.629427 (+ 1028us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:12.629440 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:12.631389 (+ 1949us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:34:12.633876 (+ 2487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:12.633884 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:12.633910 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:12.634308 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:12.634649 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:12.634652 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:12.634655 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:12.634846 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:12.635395 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:34:12.635402 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:34:12.635712 (+ 310us) client_negotiation.cc:769] Sending connection context
0810 22:34:12.635925 (+ 213us) client_negotiation.cc:240] Negotiation successful
0810 22:34:12.636306 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1400,"mutex_wait_us":346,"spinlock_wait_cycles":89856,"thread_start_us":1153,"threads_started":1}
I20260810 22:34:12.638161 24098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:12.621167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:44347 (local address 127.16.233.126:39177)
0810 22:34:12.621338 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:12.621343 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:12.622356 (+ 1013us) server_negotiation.cc:407] Connection header received
0810 22:34:12.623333 (+ 977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:12.623336 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:12.623402 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:12.623490 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:12.625252 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:12.626082 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:12.627686 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:12.628038 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:12.628142 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:12.632167 (+ 4025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:12.632175 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:12.632181 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:12.632226 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:12.634954 (+ 2728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:12.635691 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:12.635698 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:12.635703 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:12.635845 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:12.636271 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:12.636277 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:12.636282 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:12.636619 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:12.636743 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:12.637698 (+ 955us) server_negotiation.cc:299] Negotiation successful
0810 22:34:12.637887 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 22:34:12.638733 25139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:12.620640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39177 (local address 127.16.233.65:44347)
0810 22:34:12.622183 (+ 1543us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:12.622231 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:12.623104 (+ 873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:12.623703 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:12.623714 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:12.624345 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:12.625077 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:12.625093 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:12.626224 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:12.626227 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:12.627508 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:12.627522 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:12.628338 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:12.628344 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:12.628509 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:12.629315 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:12.629354 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:12.631939 (+ 2585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:34:12.635138 (+ 3199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:12.635144 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:12.635149 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:12.635515 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:12.635994 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:12.636000 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:12.636005 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:12.636131 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:12.636803 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:34:12.636833 (+ 30us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:34:12.636977 (+ 144us) client_negotiation.cc:769] Sending connection context
0810 22:34:12.637113 (+ 136us) client_negotiation.cc:240] Negotiation successful
0810 22:34:12.637342 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1327,"mutex_wait_us":250,"spinlock_wait_cycles":27264,"thread_start_us":283,"threads_started":1}
I20260810 22:34:12.639505 25134 heartbeater.cc:347] Connected to a master server at 127.16.233.124:37989
I20260810 22:34:12.639833 25134 heartbeater.cc:464] Registering TS with master...
I20260810 22:34:12.640698 25134 heartbeater.cc:511] Master 127.16.233.124:37989 requested a full tablet report, sending...
I20260810 22:34:12.641505 25135 heartbeater.cc:347] Connected to a master server at 127.16.233.126:39177
I20260810 22:34:12.641620 25135 heartbeater.cc:464] Registering TS with master...
I20260810 22:34:12.641894 25135 heartbeater.cc:511] Master 127.16.233.126:39177 requested a full tablet report, sending...
I20260810 22:34:12.642613 24937 ts_manager.cc:194] Registered new tserver with Master: 599ede616cdd412fa79af64abd77799a (127.16.233.65:39951)
I20260810 22:34:12.642995 24057 ts_manager.cc:194] Re-registered known tserver with Master: 599ede616cdd412fa79af64abd77799a (127.16.233.65:39951)
I20260810 22:34:12.644105 24057 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.233.65@KRBTEST.COM'} at 127.16.233.65:44347
I20260810 22:34:12.651249 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.66:37385
--local_ip_for_outbound_sockets=127.16.233.66
--tserver_master_addrs=127.16.233.126:39177
--webserver_port=39831
--webserver_interface=127.16.233.66
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/krb5kdc/oryx_127.16.233.66.keytab
--principal=oryx/127.16.233.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:33445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.16.233.126:39177,127.16.233.125:34439
--tserver_master_addrs=127.16.233.126:39177,127.16.233.125:34439,127.16.233.124:37989 with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 22:34:12 dist-test-slave-77v9 krb5kdc[24006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401252, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.65@KRBTEST.COM for oryx/127.16.233.125@KRBTEST.COM
I20260810 22:34:12.655743 24562 heartbeater.cc:503] Master 127.16.233.126:39177 was elected leader, sending a full tablet report...
I20260810 22:34:12.658849 24863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:12.620917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:35559 (local address 127.16.233.125:34439)
0810 22:34:12.621079 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:12.621084 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:12.623653 (+ 2569us) server_negotiation.cc:407] Connection header received
0810 22:34:12.623754 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:12.623757 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:12.623808 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:12.623907 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:12.625252 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:12.626081 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:12.649682 (+ 23601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:12.649945 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:12.650046 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:12.654802 (+ 4756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:12.654810 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:12.654816 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:12.654864 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:12.656875 (+ 2011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:12.657434 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:12.657439 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:12.657441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:12.657550 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:12.657898 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:12.657902 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:12.657904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:12.658118 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:12.658230 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:12.658556 (+ 326us) server_negotiation.cc:299] Negotiation successful
0810 22:34:12.658678 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 22:34:12.658892 25141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:12.620640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.125:34439 (local address 127.16.233.65:35559)
0810 22:34:12.623548 (+ 2908us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:12.623563 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:12.623661 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:12.623939 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:12.623942 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:12.624345 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:12.625073 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:12.625093 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:12.626225 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:12.626227 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:12.649431 (+ 23204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:12.649455 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:12.650139 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:12.650146 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:12.650313 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:12.651699 (+ 1386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:12.651721 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:12.654553 (+ 2832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:34:12.657018 (+ 2465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:12.657021 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:12.657023 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:12.657297 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:12.657682 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:12.657684 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:12.657685 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:12.657781 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:12.658236 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:34:12.658240 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:34:12.658358 (+ 118us) client_negotiation.cc:769] Sending connection context
0810 22:34:12.658478 (+ 120us) client_negotiation.cc:240] Negotiation successful
0810 22:34:12.658682 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2822,"mutex_wait_us":502,"spinlock_wait_cycles":31872,"thread_start_us":983,"threads_started":1}
I20260810 22:34:12.659651 25137 heartbeater.cc:347] Connected to a master server at 127.16.233.125:34439
I20260810 22:34:12.659756 25137 heartbeater.cc:464] Registering TS with master...
I20260810 22:34:12.660024 25137 heartbeater.cc:511] Master 127.16.233.125:34439 requested a full tablet report, sending...
I20260810 22:34:12.661317 24729 ts_manager.cc:194] Registered new tserver with Master: 599ede616cdd412fa79af64abd77799a (127.16.233.65:39951)
W20260810 22:34:12.783600 25143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:34:12.783969 25143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:34:12.784056 25143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:34:12.789618 25143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:34:12.789718 25143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:34:12.789824 25143 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.66
I20260810 22:34:12.796656 25143 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:33445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/krb5kdc/oryx_127.16.233.66.keytab
--principal=oryx/127.16.233.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.66:37385
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.233.66
--webserver_port=39831
--tserver_master_addrs=127.16.233.126:39177,127.16.233.125:34439,127.16.233.124:37989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25143
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.66
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:34:12.797945 25143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:34:12.799129 25143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:34:12.809753 25151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:12.809756 25149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:12.809756 25148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:12.810091 25143 server_base.cc:1034] running on GCE node
I20260810 22:34:12.810783 25143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:34:12.811833 25143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:34:12.813011 25143 hybrid_clock.cc:648] HybridClock initialized: now 1786401252812974 us; error 65 us; skew 500 ppm
Aug 10 22:34:12 dist-test-slave-77v9 krb5kdc[24006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401252, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:34:12.816504 25143 init.cc:377] Logged in from keytab as oryx/127.16.233.66@KRBTEST.COM (short username oryx)
I20260810 22:34:12.817711 25143 webserver.cc:460] Webserver started at http://127.16.233.66:39831/ using document root <none> and password file <none>
I20260810 22:34:12.818392 25143 fs_manager.cc:359] Metadata directory not provided
I20260810 22:34:12.818491 25143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:34:12.823170 25143 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 22:34:12.825654 25159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:12.827279 25143 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:34:12.827463 25143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-1/wal
uuid: "83e9b7216a8040b3a3418e588ecf2959"
format_stamp: "Formatted at 2026-08-10 22:34:11 on dist-test-slave-77v9"
server_key: "ddf5a5a39f27720ceef86b21a0bdbbb0"
server_key_iv: "2c7b95f0aed390311bce3452a0031e4e"
server_key_version: "encryptionkey@0"
I20260810 22:34:12.828241 25143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:34:12.844081 25143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:34:12.848163 25143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:34:12.848434 25143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:34:12.849274 25143 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:34:12.850317 25143 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:34:12.850370 25143 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:12.850441 25143 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:34:12.850479 25143 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:12.861543 25143 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.66:37385
I20260810 22:34:12.861604 25288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.66:37385 every 8 connection(s)
I20260810 22:34:12.862728 25143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
I20260810 22:34:12.871738 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 25143
I20260810 22:34:12.871958 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 24419
Aug 10 22:34:12 dist-test-slave-77v9 krb5kdc[24006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401252, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.66@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM
Aug 10 22:34:12 dist-test-slave-77v9 krb5kdc[24006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401252, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.66@KRBTEST.COM for oryx/127.16.233.124@KRBTEST.COM
I20260810 22:34:12.882190 25140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:12.864768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:46287 (local address 127.16.233.124:37989)
0810 22:34:12.864973 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:12.864979 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:12.866625 (+ 1646us) server_negotiation.cc:407] Connection header received
0810 22:34:12.867264 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:12.867268 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:12.867342 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:12.867444 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:12.869028 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:12.869662 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:12.871271 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:12.871648 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:12.871767 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:12.877643 (+ 5876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:12.877646 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:12.877648 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:12.877676 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:12.879359 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:12.880001 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:12.880005 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:12.880007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:12.880098 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:12.880494 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:12.880503 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:12.880504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:12.880783 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:12.880941 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:12.881848 (+ 907us) server_negotiation.cc:299] Negotiation successful
0810 22:34:12.881996 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260810 22:34:12.882215 24098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:12.864766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:47197 (local address 127.16.233.126:39177)
0810 22:34:12.864956 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:12.864962 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:12.866056 (+ 1094us) server_negotiation.cc:407] Connection header received
0810 22:34:12.867396 (+ 1340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:12.867399 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:12.867478 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:12.867563 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:12.869894 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:12.870677 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:12.871862 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:12.872225 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:12.872324 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:12.877628 (+ 5304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:12.877635 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:12.877641 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:12.877681 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:12.879724 (+ 2043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:12.880264 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:12.880271 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:12.880276 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:12.880381 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:12.880832 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:12.880836 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:12.880838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:12.881083 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:12.881162 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:12.881855 (+ 693us) server_negotiation.cc:299] Negotiation successful
0810 22:34:12.882022 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260810 22:34:12.882902 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.67:39107
--local_ip_for_outbound_sockets=127.16.233.67
--tserver_master_addrs=127.16.233.126:39177
--webserver_port=39629
--webserver_interface=127.16.233.67
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/krb5kdc/oryx_127.16.233.67.keytab
--principal=oryx/127.16.233.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:33445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.16.233.126:39177,127.16.233.125:34439
--tserver_master_addrs=127.16.233.126:39177,127.16.233.125:34439,127.16.233.124:37989 with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 22:34:12.883760 25293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:12.865305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39177 (local address 127.16.233.66:47197)
0810 22:34:12.865877 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:12.865934 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:12.867022 (+ 1088us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:12.867663 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:12.867670 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:12.868304 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:12.868778 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:12.868788 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:12.870785 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:12.870789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:12.871688 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:12.871708 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:12.873231 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:12.873235 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:12.873491 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:12.874290 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:12.874320 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:12.877383 (+ 3063us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:34:12.879859 (+ 2476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:12.879862 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:12.879864 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:12.880131 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:12.880500 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:12.880506 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:12.880581 (+ 75us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:12.880687 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:12.881199 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:34:12.881210 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:34:12.881527 (+ 317us) client_negotiation.cc:769] Sending connection context
0810 22:34:12.882798 (+ 1271us) client_negotiation.cc:240] Negotiation successful
0810 22:34:12.883198 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"mutex_wait_us":112,"spinlock_wait_cycles":13696,"thread_start_us":152,"threads_started":1}
I20260810 22:34:12.884075 25295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:12.865716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.124:37989 (local address 127.16.233.66:46287)
0810 22:34:12.866533 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:12.866545 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:12.867048 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:12.867665 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:12.867672 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:12.868259 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:12.868767 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:12.868781 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:12.869836 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:12.869843 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:12.871143 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:12.871152 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:12.871975 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:12.871980 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:12.873667 (+ 1687us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:12.874781 (+ 1114us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:12.874798 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:12.877383 (+ 2585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:34:12.879502 (+ 2119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:12.879509 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:12.879528 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:12.879866 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:12.880218 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:12.880220 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:12.880222 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:12.880383 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:12.881204 (+ 821us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:34:12.881213 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:34:12.881515 (+ 302us) client_negotiation.cc:769] Sending connection context
0810 22:34:12.881774 (+ 259us) client_negotiation.cc:240] Negotiation successful
0810 22:34:12.882143 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":744,"mutex_wait_us":69,"thread_start_us":84,"threads_started":1}
Aug 10 22:34:12 dist-test-slave-77v9 krb5kdc[24006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401252, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.66@KRBTEST.COM for oryx/127.16.233.125@KRBTEST.COM
I20260810 22:34:12.886122 25290 heartbeater.cc:347] Connected to a master server at 127.16.233.126:39177
I20260810 22:34:12.886577 25290 heartbeater.cc:464] Registering TS with master...
I20260810 22:34:12.887472 25290 heartbeater.cc:511] Master 127.16.233.126:39177 requested a full tablet report, sending...
I20260810 22:34:12.886122 25289 heartbeater.cc:347] Connected to a master server at 127.16.233.124:37989
I20260810 22:34:12.888382 25289 heartbeater.cc:464] Registering TS with master...
I20260810 22:34:12.888659 25289 heartbeater.cc:511] Master 127.16.233.124:37989 requested a full tablet report, sending...
I20260810 22:34:12.889571 25294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:12.865305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.125:34439 (local address 127.16.233.66:32937)
0810 22:34:12.866396 (+ 1091us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:12.866410 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:12.867048 (+ 638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:12.867896 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:12.867899 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:12.868307 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:12.868778 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:12.868788 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:12.869859 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:12.869862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:12.871160 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:12.871172 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:12.871977 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:12.871980 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:12.882605 (+ 10625us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:12.883089 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:12.883109 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:12.884695 (+ 1586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:34:12.887102 (+ 2407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:12.887107 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:12.887111 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:12.887592 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:12.888000 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:12.888003 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:12.888018 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:12.888241 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:12.888976 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:34:12.888979 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:34:12.889095 (+ 116us) client_negotiation.cc:769] Sending connection context
0810 22:34:12.889201 (+ 106us) client_negotiation.cc:240] Negotiation successful
0810 22:34:12.889377 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":990,"mutex_wait_us":10740,"thread_start_us":174,"threads_started":1}
I20260810 22:34:12.889698 24937 ts_manager.cc:194] Registered new tserver with Master: 83e9b7216a8040b3a3418e588ecf2959 (127.16.233.66:37385)
I20260810 22:34:12.890122 24057 ts_manager.cc:194] Re-registered known tserver with Master: 83e9b7216a8040b3a3418e588ecf2959 (127.16.233.66:37385)
I20260810 22:34:12.890949 24863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:12.865196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:32937 (local address 127.16.233.125:34439)
0810 22:34:12.865358 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:12.865366 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:12.866540 (+ 1174us) server_negotiation.cc:407] Connection header received
0810 22:34:12.867697 (+ 1157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:12.867701 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:12.867753 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:12.867847 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:12.868959 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:12.869709 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:12.871331 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:12.871695 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:12.871794 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:12.884929 (+ 13135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:12.884934 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:12.884940 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:12.884982 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:12.886892 (+ 1910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:12.887733 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:12.887740 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:12.887745 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:12.887869 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:12.888447 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:12.888451 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:12.888453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:12.888873 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:12.888980 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:12.890647 (+ 1667us) server_negotiation.cc:299] Negotiation successful
0810 22:34:12.890792 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 22:34:12.890919 24057 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.233.66@KRBTEST.COM'} at 127.16.233.66:47197
I20260810 22:34:12.891718 25291 heartbeater.cc:347] Connected to a master server at 127.16.233.125:34439
I20260810 22:34:12.891844 25291 heartbeater.cc:464] Registering TS with master...
I20260810 22:34:12.892539 25291 heartbeater.cc:511] Master 127.16.233.125:34439 requested a full tablet report, sending...
I20260810 22:34:12.893467 24729 ts_manager.cc:194] Registered new tserver with Master: 83e9b7216a8040b3a3418e588ecf2959 (127.16.233.66:37385)
I20260810 22:34:12.960270 25140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:12.954512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52398 (local address 127.16.233.124:37989)
0810 22:34:12.954682 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:12.954687 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:12.954868 (+ 181us) server_negotiation.cc:407] Connection header received
0810 22:34:12.955064 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:12.955071 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:12.955161 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:12.955332 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:34:12.955762 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:12.956925 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:12.958800 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:12.959559 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:12.959712 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:12.959899 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:12.959990 (+ 91us) server_negotiation.cc:299] Negotiation successful
0810 22:34:12.960113 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 22:34:12.960264 25297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:12.954407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.124:37989 (local address 127.0.0.1:52398)
0810 22:34:12.954737 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:12.954757 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:12.954934 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:12.955347 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:12.955350 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:12.955373 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:34:12.955514 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:12.955523 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:12.957197 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:12.957202 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:12.958604 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:12.958624 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:12.959717 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:12.959721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:12.959847 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:12.959878 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 22:34:12.959975 (+ 97us) client_negotiation.cc:240] Negotiation successful
0810 22:34:12.960083 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":110,"threads_started":1}
I20260810 22:34:12.961124 24948 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260810 22:34:12.961607 24986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61c3a2bed2604ebcb00d2c5647841a3c. Latest consensus state: current_term: 1 leader_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39177 } } peers { permanent_uuid: "580d354da5eb4d03a86d298eb6bc5af7" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 34439 } attrs { promote: false } } peers { permanent_uuid: "8059d406ab3242d3868f653160912ca1" member_type: NON_VOTER last_known_addr { host: "127.16.233.124" port: 37989 } attrs { promote: true } } }
I20260810 22:34:12.961773 24986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1 [sys.catalog]: This master's current role is: LEARNER
W20260810 22:34:13.028044 25296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:34:13.028409 25296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:34:13.028501 25296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:34:13.033489 25296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:34:13.033598 25296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:34:13.033716 25296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.67
I20260810 22:34:13.039441 25296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:33445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/krb5kdc/oryx_127.16.233.67.keytab
--principal=oryx/127.16.233.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.67:39107
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.233.67
--webserver_port=39629
--tserver_master_addrs=127.16.233.126:39177,127.16.233.125:34439,127.16.233.124:37989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.67
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:34:13.040777 25296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:34:13.042107 25296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:34:13.053247 25302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:13.053277 25303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:13.053629 25296 server_base.cc:1034] running on GCE node
W20260810 22:34:13.053277 25305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:13.054255 25296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:34:13.055235 25296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:34:13.056486 25296 hybrid_clock.cc:648] HybridClock initialized: now 1786401253056458 us; error 45 us; skew 500 ppm
Aug 10 22:34:13 dist-test-slave-77v9 krb5kdc[24006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401253, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:34:13.060086 25296 init.cc:377] Logged in from keytab as oryx/127.16.233.67@KRBTEST.COM (short username oryx)
I20260810 22:34:13.061329 25296 webserver.cc:460] Webserver started at http://127.16.233.67:39629/ using document root <none> and password file <none>
I20260810 22:34:13.062016 25296 fs_manager.cc:359] Metadata directory not provided
I20260810 22:34:13.062147 25296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:34:13.067108 25296 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 22:34:13.070055 25315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:13.071846 25296 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 22:34:13.072032 25296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-2/wal
uuid: "14098e2fbdf1420da6d97dd9a26616f1"
format_stamp: "Formatted at 2026-08-10 22:34:11 on dist-test-slave-77v9"
server_key: "dc6bd56a3e584d9d2a34b65bf71d2fa1"
server_key_iv: "85eee8965c601fc6a87336955fd464af"
server_key_version: "encryptionkey@0"
I20260810 22:34:13.072743 25296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:34:13.091084 25296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:34:13.095824 25296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:34:13.096092 25296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:34:13.097131 25296 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:34:13.098281 25296 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:34:13.098342 25296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:13.098392 25296 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:34:13.098456 25296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:13.109673 25296 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.67:39107
I20260810 22:34:13.109691 25443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.67:39107 every 8 connection(s)
I20260810 22:34:13.110965 25296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
I20260810 22:34:13.115264 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 25296
Aug 10 22:34:13 dist-test-slave-77v9 krb5kdc[24006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401253, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.67@KRBTEST.COM for oryx/127.16.233.125@KRBTEST.COM
Aug 10 22:34:13 dist-test-slave-77v9 krb5kdc[24006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401253, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.67@KRBTEST.COM for oryx/127.16.233.124@KRBTEST.COM
Aug 10 22:34:13 dist-test-slave-77v9 krb5kdc[24006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401253, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.67@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM
I20260810 22:34:13.132236 24863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:13.113528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:36245 (local address 127.16.233.125:34439)
0810 22:34:13.113919 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:13.113923 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:13.114773 (+ 850us) server_negotiation.cc:407] Connection header received
0810 22:34:13.116025 (+ 1252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:13.116029 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:13.116087 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:13.116144 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:13.118406 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:13.119144 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:13.122000 (+ 2856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:13.122338 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:13.122418 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:13.125770 (+ 3352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:13.125779 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:13.125786 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:13.125822 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:13.127962 (+ 2140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:13.130174 (+ 2212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:13.130178 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:13.130180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:13.130267 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:13.130742 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:13.130745 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:13.130747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:13.131015 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:13.131100 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:13.131890 (+ 790us) server_negotiation.cc:299] Negotiation successful
0810 22:34:13.132026 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":311}
I20260810 22:34:13.132699 25458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:13.118029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52404 (local address 127.16.233.124:37989)
0810 22:34:13.118307 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:13.118314 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:13.120091 (+ 1777us) server_negotiation.cc:407] Connection header received
0810 22:34:13.120136 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:13.120139 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:13.120190 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:13.120302 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:13.122128 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:13.122896 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:13.124461 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:13.124896 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:13.125001 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:13.127971 (+ 2970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:13.127976 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:13.127980 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:13.128011 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:13.130623 (+ 2612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:13.131102 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:13.131105 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:13.131109 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:13.131200 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:13.131538 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:13.131542 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:13.131543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:13.131753 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:13.131879 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:13.132297 (+ 418us) server_negotiation.cc:299] Negotiation successful
0810 22:34:13.132536 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":77,"threads_started":1}
I20260810 22:34:13.134421 25448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:13.113825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.125:34439 (local address 127.16.233.67:36245)
0810 22:34:13.114468 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:13.114526 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:13.115764 (+ 1238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:13.116579 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:13.116589 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:13.117575 (+ 986us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:13.118248 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:13.118259 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:13.119357 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:13.119360 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:13.121851 (+ 2491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:13.121861 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:13.122527 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:13.122530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:13.122775 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:13.123495 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:13.123517 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:13.125565 (+ 2048us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:34:13.128141 (+ 2576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:13.128148 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:13.128167 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:13.128562 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:13.130393 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:13.130397 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:13.130399 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:13.130623 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:13.131174 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:34:13.131182 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:34:13.131562 (+ 380us) client_negotiation.cc:769] Sending connection context
0810 22:34:13.132665 (+ 1103us) client_negotiation.cc:240] Negotiation successful
0810 22:34:13.133320 (+ 655us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"mutex_wait_us":1883,"thread_start_us":207,"threads_started":1}
I20260810 22:34:13.134836 25140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:13.113658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:46177 (local address 127.16.233.124:37989)
0810 22:34:13.114018 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:13.114023 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:13.115153 (+ 1130us) server_negotiation.cc:407] Connection header received
0810 22:34:13.116033 (+ 880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:13.116037 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:13.116097 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:13.116155 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:13.118290 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:13.119153 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:13.121155 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:13.121489 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:13.121593 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:13.128902 (+ 7309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:13.128906 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:13.128908 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:13.128940 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:13.131445 (+ 2505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:13.132332 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:13.132336 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:13.132341 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:13.132434 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:13.133830 (+ 1396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:13.133834 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:13.133835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:13.134075 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:13.134169 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:13.134582 (+ 413us) server_negotiation.cc:299] Negotiation successful
0810 22:34:13.134707 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284}
I20260810 22:34:13.134852 25449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:13.113825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.124:37989 (local address 127.16.233.67:46177)
0810 22:34:13.115016 (+ 1191us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:13.115035 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:13.115769 (+ 734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:13.116581 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:13.116590 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:13.117319 (+ 729us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:13.118101 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:13.118122 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:13.119357 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:13.119361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:13.121030 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:13.121045 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:13.121668 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:13.121672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:13.123892 (+ 2220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:13.125498 (+ 1606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:13.125515 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:13.128714 (+ 3199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:34:13.131660 (+ 2946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:13.131665 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:13.131670 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:13.131953 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:13.132799 (+ 846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:13.132805 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:13.132836 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:13.132928 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:13.134236 (+ 1308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:34:13.134240 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:34:13.134345 (+ 105us) client_negotiation.cc:769] Sending connection context
0810 22:34:13.134456 (+ 111us) client_negotiation.cc:240] Negotiation successful
0810 22:34:13.134608 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1077,"mutex_wait_us":2224,"spinlock_wait_cycles":86272,"thread_start_us":207,"threads_started":1}
I20260810 22:34:13.135725 25456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:13.117544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45646 (local address 127.16.233.126:39177)
0810 22:34:13.117812 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:13.117818 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:13.119517 (+ 1699us) server_negotiation.cc:407] Connection header received
0810 22:34:13.119725 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:13.119730 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:13.119791 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:13.120008 (+ 217us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:13.121042 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:13.121812 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:13.126095 (+ 4283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:13.126524 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:13.126630 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:13.128222 (+ 1592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:13.128226 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:13.128229 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:13.128276 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:13.131134 (+ 2858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:13.132099 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:13.132103 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:13.132105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:13.132203 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:13.133070 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:13.133074 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:13.133076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:13.133343 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:13.133466 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:13.135384 (+ 1918us) server_negotiation.cc:299] Negotiation successful
0810 22:34:13.135530 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":86,"threads_started":1}
I20260810 22:34:13.135979 25444 heartbeater.cc:347] Connected to a master server at 127.16.233.124:37989
I20260810 22:34:13.136152 25446 heartbeater.cc:347] Connected to a master server at 127.16.233.125:34439
I20260810 22:34:13.136221 25444 heartbeater.cc:464] Registering TS with master...
I20260810 22:34:13.136268 25446 heartbeater.cc:464] Registering TS with master...
I20260810 22:34:13.136866 25444 heartbeater.cc:511] Master 127.16.233.124:37989 requested a full tablet report, sending...
I20260810 22:34:13.137303 24098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:13.113384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:41219 (local address 127.16.233.126:39177)
0810 22:34:13.113811 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:13.113816 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:13.115434 (+ 1618us) server_negotiation.cc:407] Connection header received
0810 22:34:13.116025 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:13.116029 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:13.116087 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:13.116144 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:13.118439 (+ 2295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:13.119202 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:13.123662 (+ 4460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:13.123999 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:13.124074 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:13.127973 (+ 3899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:13.127977 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:13.127981 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:13.128013 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:13.132284 (+ 4271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:13.133446 (+ 1162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:13.133449 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:13.133451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:13.133583 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:13.135788 (+ 2205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:13.135792 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:13.135793 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:13.136018 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:13.136107 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:13.137016 (+ 909us) server_negotiation.cc:299] Negotiation successful
0810 22:34:13.137146 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323}
I20260810 22:34:13.137465 25446 heartbeater.cc:511] Master 127.16.233.125:34439 requested a full tablet report, sending...
I20260810 22:34:13.138273 24729 ts_manager.cc:194] Registered new tserver with Master: 14098e2fbdf1420da6d97dd9a26616f1 (127.16.233.67:39107)
I20260810 22:34:13.138523 25450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:13.113825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39177 (local address 127.16.233.67:41219)
0810 22:34:13.115333 (+ 1508us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:13.115350 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:13.115764 (+ 414us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:13.116579 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:13.116589 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:13.117319 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:13.118101 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:13.118122 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:13.119364 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:13.119366 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:13.123528 (+ 4162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:13.123537 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:13.124534 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:13.124540 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:13.124634 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:13.126165 (+ 1531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:13.126181 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:13.127791 (+ 1610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:34:13.132496 (+ 4705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:13.132502 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:13.132507 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:13.132773 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:13.133758 (+ 985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:13.133762 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:13.133766 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:13.133861 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:13.136687 (+ 2826us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:34:13.136689 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:34:13.136763 (+ 74us) client_negotiation.cc:769] Sending connection context
0810 22:34:13.137518 (+ 755us) client_negotiation.cc:240] Negotiation successful
0810 22:34:13.138004 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1403,"mutex_wait_us":3333,"thread_start_us":122,"threads_started":1}
I20260810 22:34:13.138787 24937 ts_manager.cc:194] Registered new tserver with Master: 14098e2fbdf1420da6d97dd9a26616f1 (127.16.233.67:39107)
I20260810 22:34:13.139338 25445 heartbeater.cc:347] Connected to a master server at 127.16.233.126:39177
I20260810 22:34:13.139436 25445 heartbeater.cc:464] Registering TS with master...
I20260810 22:34:13.139623 25445 heartbeater.cc:511] Master 127.16.233.126:39177 requested a full tablet report, sending...
I20260810 22:34:13.139909 25459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:13.118699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50952 (local address 127.16.233.125:34439)
0810 22:34:13.119262 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:13.119268 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:13.120603 (+ 1335us) server_negotiation.cc:407] Connection header received
0810 22:34:13.120701 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:13.120704 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:13.120757 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:13.120909 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:13.122428 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:13.123319 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:13.124580 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:13.125117 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:13.125210 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:13.129797 (+ 4587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:13.129804 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:13.129808 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:13.129838 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:13.134839 (+ 5001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:13.136918 (+ 2079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:13.136924 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:13.136929 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:13.137704 (+ 775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:13.138894 (+ 1190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:13.138898 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:13.138900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:13.139148 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:13.139264 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:13.139624 (+ 360us) server_negotiation.cc:299] Negotiation successful
0810 22:34:13.139756 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":483,"thread_start_us":93,"threads_started":1}
I20260810 22:34:13.140266 24057 ts_manager.cc:194] Re-registered known tserver with Master: 14098e2fbdf1420da6d97dd9a26616f1 (127.16.233.67:39107)
I20260810 22:34:13.141090 24057 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.233.67@KRBTEST.COM'} at 127.16.233.67:41219
I20260810 22:34:13.139415 24056 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45646:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 22:34:13.143164 24056 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:34:13.145995 24897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61c3a2bed2604ebcb00d2c5647841a3c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39177 } } peers { permanent_uuid: "580d354da5eb4d03a86d298eb6bc5af7" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 34439 } attrs { promote: false } } peers { permanent_uuid: "8059d406ab3242d3868f653160912ca1" member_type: NON_VOTER last_known_addr { host: "127.16.233.124" port: 37989 } attrs { promote: true } } }
I20260810 22:34:13.146144 24897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61c3a2bed2604ebcb00d2c5647841a3c [sys.catalog]: This master's current role is: LEADER
I20260810 22:34:13.147917 25463 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 61c3a2bed2604ebcb00d2c5647841a3c: attempting to promote NON_VOTER 8059d406ab3242d3868f653160912ca1 to VOTER
I20260810 22:34:13.148281 25463 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61c3a2bed2604ebcb00d2c5647841a3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39177 } } peers { permanent_uuid: "580d354da5eb4d03a86d298eb6bc5af7" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 34439 } attrs { promote: false } } peers { permanent_uuid: "8059d406ab3242d3868f653160912ca1" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 37989 } attrs { promote: false } }
I20260810 22:34:13.149559 24948 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1 [term 1 LEARNER]: Refusing update from remote peer 61c3a2bed2604ebcb00d2c5647841a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260810 22:34:13.149648 24738 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7 [term 1 FOLLOWER]: Refusing update from remote peer 61c3a2bed2604ebcb00d2c5647841a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260810 22:34:13.150101 25465 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 61c3a2bed2604ebcb00d2c5647841a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8059d406ab3242d3868f653160912ca1" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 37989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260810 22:34:13.150307 24897 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 61c3a2bed2604ebcb00d2c5647841a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "580d354da5eb4d03a86d298eb6bc5af7" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 34439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260810 22:34:13.152439 25463 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 61c3a2bed2604ebcb00d2c5647841a3c [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 8059d406ab3242d3868f653160912ca1 (127.16.233.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39177 } } peers { permanent_uuid: "580d354da5eb4d03a86d298eb6bc5af7" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 34439 } attrs { promote: false } } peers { permanent_uuid: "8059d406ab3242d3868f653160912ca1" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 37989 } attrs { promote: false } } }
I20260810 22:34:13.152926 24948 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 8059d406ab3242d3868f653160912ca1 (127.16.233.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39177 } } peers { permanent_uuid: "580d354da5eb4d03a86d298eb6bc5af7" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 34439 } attrs { promote: false } } peers { permanent_uuid: "8059d406ab3242d3868f653160912ca1" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 37989 } attrs { promote: false } } }
I20260810 22:34:13.153033 24738 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 8059d406ab3242d3868f653160912ca1 (127.16.233.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39177 } } peers { permanent_uuid: "580d354da5eb4d03a86d298eb6bc5af7" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 34439 } attrs { promote: false } } peers { permanent_uuid: "8059d406ab3242d3868f653160912ca1" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 37989 } attrs { promote: false } } }
I20260810 22:34:13.154585 24986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39177 } } peers { permanent_uuid: "580d354da5eb4d03a86d298eb6bc5af7" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 34439 } attrs { promote: false } } peers { permanent_uuid: "8059d406ab3242d3868f653160912ca1" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 37989 } attrs { promote: false } } }
I20260810 22:34:13.154721 24986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 22:34:13.154855 24897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61c3a2bed2604ebcb00d2c5647841a3c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39177 } } peers { permanent_uuid: "580d354da5eb4d03a86d298eb6bc5af7" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 34439 } attrs { promote: false } } peers { permanent_uuid: "8059d406ab3242d3868f653160912ca1" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 37989 } attrs { promote: false } } }
I20260810 22:34:13.154975 24897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61c3a2bed2604ebcb00d2c5647841a3c [sys.catalog]: This master's current role is: LEADER
I20260810 22:34:13.155731 25475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39177 } } peers { permanent_uuid: "580d354da5eb4d03a86d298eb6bc5af7" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 34439 } attrs { promote: false } } peers { permanent_uuid: "8059d406ab3242d3868f653160912ca1" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 37989 } attrs { promote: false } } }
I20260810 22:34:13.156234 25475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 22:34:13.156224 25465 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61c3a2bed2604ebcb00d2c5647841a3c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39177 } } peers { permanent_uuid: "580d354da5eb4d03a86d298eb6bc5af7" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 34439 } attrs { promote: false } } peers { permanent_uuid: "8059d406ab3242d3868f653160912ca1" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 37989 } attrs { promote: false } } }
I20260810 22:34:13.156325 25465 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61c3a2bed2604ebcb00d2c5647841a3c [sys.catalog]: This master's current role is: LEADER
I20260810 22:34:13.156492 25463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61c3a2bed2604ebcb00d2c5647841a3c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39177 } } peers { permanent_uuid: "580d354da5eb4d03a86d298eb6bc5af7" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 34439 } attrs { promote: false } } peers { permanent_uuid: "8059d406ab3242d3868f653160912ca1" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 37989 } attrs { promote: false } } }
I20260810 22:34:13.156579 25463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61c3a2bed2604ebcb00d2c5647841a3c [sys.catalog]: This master's current role is: LEADER
I20260810 22:34:13.166712 25478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:13.160855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42588 (local address 127.16.233.66:37385)
0810 22:34:13.162343 (+ 1488us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:13.162352 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:13.162373 (+ 21us) server_negotiation.cc:407] Connection header received
0810 22:34:13.162460 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:13.162464 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:13.162602 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:13.162767 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:34:13.163010 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:13.164051 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:13.165474 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:13.166046 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:13.166177 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:13.166344 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:13.166466 (+ 122us) server_negotiation.cc:299] Negotiation successful
0810 22:34:13.166567 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1390,"thread_start_us":132,"threads_started":1}
I20260810 22:34:13.166698 25297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:13.160374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.66:37385 (local address 127.0.0.1:42588)
0810 22:34:13.160528 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:13.160548 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:13.160639 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:13.162745 (+ 2106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:13.162748 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:13.162775 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:34:13.162866 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:13.162875 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:13.164170 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:13.164174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:13.165339 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:13.165355 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:13.166186 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:13.166189 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:13.166310 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:13.166334 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 22:34:13.166419 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 22:34:13.166539 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75,"mutex_wait_us":67}
I20260810 22:34:13.167048 25477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:13.160555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.65:39951 (local address 127.0.0.1:60500)
0810 22:34:13.161750 (+ 1195us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:13.161767 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:13.161862 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:13.162444 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:13.162447 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:13.162465 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:34:13.162587 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:13.162594 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:13.163938 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:13.163941 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:13.165306 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:13.165318 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:13.166610 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:13.166614 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:13.166707 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:13.166730 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 22:34:13.166812 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 22:34:13.166885 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1133,"mutex_wait_us":39,"thread_start_us":105,"threads_started":1}
I20260810 22:34:13.167582 25481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:13.161405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60500 (local address 127.16.233.65:39951)
0810 22:34:13.161901 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:13.161911 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:13.161939 (+ 28us) server_negotiation.cc:407] Connection header received
0810 22:34:13.162028 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:13.162034 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:13.162277 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:13.162481 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:34:13.162695 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:13.163793 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:13.165447 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:13.166247 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:13.166379 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:13.166541 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:13.167292 (+ 751us) server_negotiation.cc:299] Negotiation successful
0810 22:34:13.167405 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":93,"threads_started":1}
I20260810 22:34:13.168671 25476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:13.160866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.67:39107 (local address 127.0.0.1:55134)
0810 22:34:13.163015 (+ 2149us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:13.163034 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:13.163140 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:13.163782 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:13.163786 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:13.163804 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:34:13.163915 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:13.163922 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:13.165051 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:13.165055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:13.166234 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:13.166245 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:13.168253 (+ 2008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:13.168257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:13.168362 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:13.168387 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 22:34:13.168472 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 22:34:13.168546 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2083,"mutex_wait_us":24,"thread_start_us":120,"threads_started":1}
I20260810 22:34:13.168735 25480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:13.161290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55134 (local address 127.16.233.67:39107)
0810 22:34:13.162111 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:13.162122 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:13.163125 (+ 1003us) server_negotiation.cc:407] Connection header received
0810 22:34:13.163247 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:13.163252 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:13.163465 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:13.163616 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:34:13.164057 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:13.164935 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:13.166391 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:13.168117 (+ 1726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:13.168236 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:13.168395 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:13.168500 (+ 105us) server_negotiation.cc:299] Negotiation successful
0810 22:34:13.168595 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":725,"thread_start_us":138,"threads_started":1}
I20260810 22:34:13.170284 25062 tablet_service.cc:1467] Processing CreateTablet for tablet 7184e4ee10824392b06d163249c4e38b (DEFAULT_TABLE table=test-table [id=29af7bb0a6d54f09ab91a5882400d56f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:34:13.170274 25213 tablet_service.cc:1467] Processing CreateTablet for tablet 7184e4ee10824392b06d163249c4e38b (DEFAULT_TABLE table=test-table [id=29af7bb0a6d54f09ab91a5882400d56f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:34:13.171391 25062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7184e4ee10824392b06d163249c4e38b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:34:13.171391 25213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7184e4ee10824392b06d163249c4e38b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:34:13.176095 25371 tablet_service.cc:1467] Processing CreateTablet for tablet 7184e4ee10824392b06d163249c4e38b (DEFAULT_TABLE table=test-table [id=29af7bb0a6d54f09ab91a5882400d56f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:34:13.177438 25371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7184e4ee10824392b06d163249c4e38b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:34:13.179107 25483 tablet_bootstrap.cc:492] T 7184e4ee10824392b06d163249c4e38b P 83e9b7216a8040b3a3418e588ecf2959: Bootstrap starting.
I20260810 22:34:13.181511 25483 tablet_bootstrap.cc:654] T 7184e4ee10824392b06d163249c4e38b P 83e9b7216a8040b3a3418e588ecf2959: Neither blocks nor log segments found. Creating new log.
I20260810 22:34:13.182665 25483 log.cc:824] T 7184e4ee10824392b06d163249c4e38b P 83e9b7216a8040b3a3418e588ecf2959: Log is configured to *not* fsync() on all Append() calls
I20260810 22:34:13.184890 25483 tablet_bootstrap.cc:492] T 7184e4ee10824392b06d163249c4e38b P 83e9b7216a8040b3a3418e588ecf2959: No bootstrap required, opened a new log
I20260810 22:34:13.185115 25483 ts_tablet_manager.cc:1397] T 7184e4ee10824392b06d163249c4e38b P 83e9b7216a8040b3a3418e588ecf2959: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 22:34:13.185454 25486 tablet_bootstrap.cc:492] T 7184e4ee10824392b06d163249c4e38b P 14098e2fbdf1420da6d97dd9a26616f1: Bootstrap starting.
I20260810 22:34:13.186075 25484 tablet_bootstrap.cc:492] T 7184e4ee10824392b06d163249c4e38b P 599ede616cdd412fa79af64abd77799a: Bootstrap starting.
I20260810 22:34:13.187788 25484 tablet_bootstrap.cc:654] T 7184e4ee10824392b06d163249c4e38b P 599ede616cdd412fa79af64abd77799a: Neither blocks nor log segments found. Creating new log.
I20260810 22:34:13.187790 25486 tablet_bootstrap.cc:654] T 7184e4ee10824392b06d163249c4e38b P 14098e2fbdf1420da6d97dd9a26616f1: Neither blocks nor log segments found. Creating new log.
I20260810 22:34:13.188499 25484 log.cc:824] T 7184e4ee10824392b06d163249c4e38b P 599ede616cdd412fa79af64abd77799a: Log is configured to *not* fsync() on all Append() calls
I20260810 22:34:13.188621 25486 log.cc:824] T 7184e4ee10824392b06d163249c4e38b P 14098e2fbdf1420da6d97dd9a26616f1: Log is configured to *not* fsync() on all Append() calls
I20260810 22:34:13.190150 25483 raft_consensus.cc:348] T 7184e4ee10824392b06d163249c4e38b P 83e9b7216a8040b3a3418e588ecf2959 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e9b7216a8040b3a3418e588ecf2959" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 37385 } } peers { permanent_uuid: "599ede616cdd412fa79af64abd77799a" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 39951 } } peers { permanent_uuid: "14098e2fbdf1420da6d97dd9a26616f1" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 39107 } }
I20260810 22:34:13.190444 25483 raft_consensus.cc:374] T 7184e4ee10824392b06d163249c4e38b P 83e9b7216a8040b3a3418e588ecf2959 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:34:13.190505 25483 raft_consensus.cc:729] T 7184e4ee10824392b06d163249c4e38b P 83e9b7216a8040b3a3418e588ecf2959 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e9b7216a8040b3a3418e588ecf2959, State: Initialized, Role: FOLLOWER
I20260810 22:34:13.190969 25484 tablet_bootstrap.cc:492] T 7184e4ee10824392b06d163249c4e38b P 599ede616cdd412fa79af64abd77799a: No bootstrap required, opened a new log
I20260810 22:34:13.190984 25483 consensus_queue.cc:260] T 7184e4ee10824392b06d163249c4e38b P 83e9b7216a8040b3a3418e588ecf2959 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e9b7216a8040b3a3418e588ecf2959" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 37385 } } peers { permanent_uuid: "599ede616cdd412fa79af64abd77799a" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 39951 } } peers { permanent_uuid: "14098e2fbdf1420da6d97dd9a26616f1" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 39107 } }
I20260810 22:34:13.191128 25486 tablet_bootstrap.cc:492] T 7184e4ee10824392b06d163249c4e38b P 14098e2fbdf1420da6d97dd9a26616f1: No bootstrap required, opened a new log
I20260810 22:34:13.191210 25484 ts_tablet_manager.cc:1397] T 7184e4ee10824392b06d163249c4e38b P 599ede616cdd412fa79af64abd77799a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 22:34:13.191303 25486 ts_tablet_manager.cc:1397] T 7184e4ee10824392b06d163249c4e38b P 14098e2fbdf1420da6d97dd9a26616f1: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 22:34:13.192056 25483 ts_tablet_manager.cc:1428] T 7184e4ee10824392b06d163249c4e38b P 83e9b7216a8040b3a3418e588ecf2959: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260810 22:34:13.192426 25290 heartbeater.cc:503] Master 127.16.233.126:39177 was elected leader, sending a full tablet report...
I20260810 22:34:13.195186 25484 raft_consensus.cc:348] T 7184e4ee10824392b06d163249c4e38b P 599ede616cdd412fa79af64abd77799a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e9b7216a8040b3a3418e588ecf2959" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 37385 } } peers { permanent_uuid: "599ede616cdd412fa79af64abd77799a" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 39951 } } peers { permanent_uuid: "14098e2fbdf1420da6d97dd9a26616f1" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 39107 } }
I20260810 22:34:13.195465 25484 raft_consensus.cc:374] T 7184e4ee10824392b06d163249c4e38b P 599ede616cdd412fa79af64abd77799a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:34:13.195523 25484 raft_consensus.cc:729] T 7184e4ee10824392b06d163249c4e38b P 599ede616cdd412fa79af64abd77799a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 599ede616cdd412fa79af64abd77799a, State: Initialized, Role: FOLLOWER
I20260810 22:34:13.196301 25484 consensus_queue.cc:260] T 7184e4ee10824392b06d163249c4e38b P 599ede616cdd412fa79af64abd77799a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e9b7216a8040b3a3418e588ecf2959" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 37385 } } peers { permanent_uuid: "599ede616cdd412fa79af64abd77799a" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 39951 } } peers { permanent_uuid: "14098e2fbdf1420da6d97dd9a26616f1" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 39107 } }
I20260810 22:34:13.197427 25484 ts_tablet_manager.cc:1428] T 7184e4ee10824392b06d163249c4e38b P 599ede616cdd412fa79af64abd77799a: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 22:34:13.197791 25135 heartbeater.cc:503] Master 127.16.233.126:39177 was elected leader, sending a full tablet report...
I20260810 22:34:13.198573 25486 raft_consensus.cc:348] T 7184e4ee10824392b06d163249c4e38b P 14098e2fbdf1420da6d97dd9a26616f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e9b7216a8040b3a3418e588ecf2959" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 37385 } } peers { permanent_uuid: "599ede616cdd412fa79af64abd77799a" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 39951 } } peers { permanent_uuid: "14098e2fbdf1420da6d97dd9a26616f1" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 39107 } }
I20260810 22:34:13.198827 25486 raft_consensus.cc:374] T 7184e4ee10824392b06d163249c4e38b P 14098e2fbdf1420da6d97dd9a26616f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:34:13.198882 25486 raft_consensus.cc:729] T 7184e4ee10824392b06d163249c4e38b P 14098e2fbdf1420da6d97dd9a26616f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14098e2fbdf1420da6d97dd9a26616f1, State: Initialized, Role: FOLLOWER
I20260810 22:34:13.199394 25486 consensus_queue.cc:260] T 7184e4ee10824392b06d163249c4e38b P 14098e2fbdf1420da6d97dd9a26616f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e9b7216a8040b3a3418e588ecf2959" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 37385 } } peers { permanent_uuid: "599ede616cdd412fa79af64abd77799a" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 39951 } } peers { permanent_uuid: "14098e2fbdf1420da6d97dd9a26616f1" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 39107 } }
I20260810 22:34:13.200353 25445 heartbeater.cc:503] Master 127.16.233.126:39177 was elected leader, sending a full tablet report...
I20260810 22:34:13.201449 25486 ts_tablet_manager.cc:1428] T 7184e4ee10824392b06d163249c4e38b P 14098e2fbdf1420da6d97dd9a26616f1: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
W20260810 22:34:13.362591 25447 tablet.cc:2369] T 7184e4ee10824392b06d163249c4e38b P 14098e2fbdf1420da6d97dd9a26616f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 22:34:13.365093 25292 tablet.cc:2369] T 7184e4ee10824392b06d163249c4e38b P 83e9b7216a8040b3a3418e588ecf2959: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 22:34:13.370648 25138 tablet.cc:2369] T 7184e4ee10824392b06d163249c4e38b P 599ede616cdd412fa79af64abd77799a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:34:13.426343 25490 raft_consensus.cc:482] T 7184e4ee10824392b06d163249c4e38b P 599ede616cdd412fa79af64abd77799a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:34:13.426728 25490 raft_consensus.cc:504] T 7184e4ee10824392b06d163249c4e38b P 599ede616cdd412fa79af64abd77799a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e9b7216a8040b3a3418e588ecf2959" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 37385 } } peers { permanent_uuid: "599ede616cdd412fa79af64abd77799a" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 39951 } } peers { permanent_uuid: "14098e2fbdf1420da6d97dd9a26616f1" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 39107 } }
I20260810 22:34:13.428488 25490 leader_election.cc:288] T 7184e4ee10824392b06d163249c4e38b P 599ede616cdd412fa79af64abd77799a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83e9b7216a8040b3a3418e588ecf2959 (127.16.233.66:37385), 14098e2fbdf1420da6d97dd9a26616f1 (127.16.233.67:39107)
I20260810 22:34:13.434823 25494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:13.429250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.67:39107 (local address 127.16.233.65:57417)
0810 22:34:13.429640 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:13.429657 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:13.429771 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:13.430091 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:13.430094 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:13.430112 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:34:13.430342 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:13.430352 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:13.431517 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:13.431521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:13.432947 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:13.432961 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:13.434073 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:13.434077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:13.434327 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:13.434349 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 22:34:13.434429 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 22:34:13.434581 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":107,"thread_start_us":118,"threads_started":1}
I20260810 22:34:13.434826 25492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:13.428738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.66:37385 (local address 127.16.233.65:52845)
0810 22:34:13.429225 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:13.429243 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:13.429390 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:13.429915 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:13.429920 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:13.429937 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:34:13.430128 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:13.430137 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:13.431255 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:13.431258 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:13.432994 (+ 1736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:13.433010 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:13.434040 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:13.434045 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:13.434157 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:13.434335 (+ 178us) client_negotiation.cc:769] Sending connection context
0810 22:34:13.434432 (+ 97us) client_negotiation.cc:240] Negotiation successful
0810 22:34:13.434581 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"mutex_wait_us":209,"thread_start_us":143,"threads_started":1}
I20260810 22:34:13.435443 25478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:13.428971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:52845 (local address 127.16.233.66:37385)
0810 22:34:13.429141 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:13.429147 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:13.429364 (+ 217us) server_negotiation.cc:407] Connection header received
0810 22:34:13.429558 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:13.429561 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:13.429648 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:13.429753 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:34:13.430287 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:13.431080 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:13.433206 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:13.433890 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:13.434047 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:13.434104 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:13.434611 (+ 507us) server_negotiation.cc:299] Negotiation successful
0810 22:34:13.435334 (+ 723us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260810 22:34:13.435775 25480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:13.429647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:57417 (local address 127.16.233.67:39107)
0810 22:34:13.429811 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:13.429816 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:13.429861 (+ 45us) server_negotiation.cc:407] Connection header received
0810 22:34:13.429934 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:13.429937 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:13.429988 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:13.430065 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:34:13.430500 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:13.431315 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:13.433110 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:13.433799 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:13.433908 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:13.433965 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:13.434706 (+ 741us) server_negotiation.cc:299] Negotiation successful
0810 22:34:13.435645 (+ 939us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260810 22:34:13.436367 25394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7184e4ee10824392b06d163249c4e38b" candidate_uuid: "599ede616cdd412fa79af64abd77799a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14098e2fbdf1420da6d97dd9a26616f1" is_pre_election: true
I20260810 22:34:13.436345 25235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7184e4ee10824392b06d163249c4e38b" candidate_uuid: "599ede616cdd412fa79af64abd77799a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e9b7216a8040b3a3418e588ecf2959" is_pre_election: true
I20260810 22:34:13.436791 25394 raft_consensus.cc:2454] T 7184e4ee10824392b06d163249c4e38b P 14098e2fbdf1420da6d97dd9a26616f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 599ede616cdd412fa79af64abd77799a in term 0.
I20260810 22:34:13.436877 25235 raft_consensus.cc:2454] T 7184e4ee10824392b06d163249c4e38b P 83e9b7216a8040b3a3418e588ecf2959 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 599ede616cdd412fa79af64abd77799a in term 0.
I20260810 22:34:13.437425 25015 leader_election.cc:302] T 7184e4ee10824392b06d163249c4e38b P 599ede616cdd412fa79af64abd77799a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 14098e2fbdf1420da6d97dd9a26616f1, 599ede616cdd412fa79af64abd77799a, 83e9b7216a8040b3a3418e588ecf2959; no voters:
I20260810 22:34:13.437817 25490 raft_consensus.cc:2781] T 7184e4ee10824392b06d163249c4e38b P 599ede616cdd412fa79af64abd77799a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:34:13.437947 25490 raft_consensus.cc:482] T 7184e4ee10824392b06d163249c4e38b P 599ede616cdd412fa79af64abd77799a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:34:13.438011 25490 raft_consensus.cc:3037] T 7184e4ee10824392b06d163249c4e38b P 599ede616cdd412fa79af64abd77799a [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:34:13.439189 25490 raft_consensus.cc:504] T 7184e4ee10824392b06d163249c4e38b P 599ede616cdd412fa79af64abd77799a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e9b7216a8040b3a3418e588ecf2959" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 37385 } } peers { permanent_uuid: "599ede616cdd412fa79af64abd77799a" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 39951 } } peers { permanent_uuid: "14098e2fbdf1420da6d97dd9a26616f1" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 39107 } }
I20260810 22:34:13.439597 25490 leader_election.cc:288] T 7184e4ee10824392b06d163249c4e38b P 599ede616cdd412fa79af64abd77799a [CANDIDATE]: Term 1 election: Requested vote from peers 83e9b7216a8040b3a3418e588ecf2959 (127.16.233.66:37385), 14098e2fbdf1420da6d97dd9a26616f1 (127.16.233.67:39107)
I20260810 22:34:13.440131 25394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7184e4ee10824392b06d163249c4e38b" candidate_uuid: "599ede616cdd412fa79af64abd77799a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14098e2fbdf1420da6d97dd9a26616f1"
I20260810 22:34:13.440160 25235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7184e4ee10824392b06d163249c4e38b" candidate_uuid: "599ede616cdd412fa79af64abd77799a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e9b7216a8040b3a3418e588ecf2959"
I20260810 22:34:13.440304 25235 raft_consensus.cc:3037] T 7184e4ee10824392b06d163249c4e38b P 83e9b7216a8040b3a3418e588ecf2959 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:34:13.440302 25394 raft_consensus.cc:3037] T 7184e4ee10824392b06d163249c4e38b P 14098e2fbdf1420da6d97dd9a26616f1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:34:13.441747 25235 raft_consensus.cc:2454] T 7184e4ee10824392b06d163249c4e38b P 83e9b7216a8040b3a3418e588ecf2959 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 599ede616cdd412fa79af64abd77799a in term 1.
I20260810 22:34:13.441751 25394 raft_consensus.cc:2454] T 7184e4ee10824392b06d163249c4e38b P 14098e2fbdf1420da6d97dd9a26616f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 599ede616cdd412fa79af64abd77799a in term 1.
I20260810 22:34:13.442279 25013 leader_election.cc:302] T 7184e4ee10824392b06d163249c4e38b P 599ede616cdd412fa79af64abd77799a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14098e2fbdf1420da6d97dd9a26616f1, 599ede616cdd412fa79af64abd77799a; no voters:
I20260810 22:34:13.442541 25490 raft_consensus.cc:2781] T 7184e4ee10824392b06d163249c4e38b P 599ede616cdd412fa79af64abd77799a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:34:13.442795 25490 raft_consensus.cc:686] T 7184e4ee10824392b06d163249c4e38b P 599ede616cdd412fa79af64abd77799a [term 1 LEADER]: Becoming Leader. State: Replica: 599ede616cdd412fa79af64abd77799a, State: Running, Role: LEADER
I20260810 22:34:13.443208 25490 consensus_queue.cc:237] T 7184e4ee10824392b06d163249c4e38b P 599ede616cdd412fa79af64abd77799a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e9b7216a8040b3a3418e588ecf2959" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 37385 } } peers { permanent_uuid: "599ede616cdd412fa79af64abd77799a" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 39951 } } peers { permanent_uuid: "14098e2fbdf1420da6d97dd9a26616f1" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 39107 } }
I20260810 22:34:13.447132 24056 catalog_manager.cc:5495] T 7184e4ee10824392b06d163249c4e38b P 599ede616cdd412fa79af64abd77799a reported cstate change: term changed from 0 to 1, leader changed from <none> to 599ede616cdd412fa79af64abd77799a (127.16.233.65). New cstate: current_term: 1 leader_uuid: "599ede616cdd412fa79af64abd77799a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83e9b7216a8040b3a3418e588ecf2959" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 37385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "599ede616cdd412fa79af64abd77799a" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 39951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14098e2fbdf1420da6d97dd9a26616f1" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 39107 } health_report { overall_health: UNKNOWN } } }
I20260810 22:34:13.468752 25481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:13.464952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60508 (local address 127.16.233.65:39951)
0810 22:34:13.465105 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:13.465113 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:13.465131 (+ 18us) server_negotiation.cc:407] Connection header received
0810 22:34:13.465228 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:13.465231 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:13.465287 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:13.465407 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:34:13.465689 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:13.466215 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:13.467141 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:13.467369 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:13.467510 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:13.467849 (+ 339us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:34:13.468309 (+ 460us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:34:13.468447 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:13.468523 (+ 76us) server_negotiation.cc:299] Negotiation successful
0810 22:34:13.468592 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 22:34:13.475576 25394 raft_consensus.cc:1261] T 7184e4ee10824392b06d163249c4e38b P 14098e2fbdf1420da6d97dd9a26616f1 [term 1 FOLLOWER]: Refusing update from remote peer 599ede616cdd412fa79af64abd77799a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:34:13.475576 25235 raft_consensus.cc:1261] T 7184e4ee10824392b06d163249c4e38b P 83e9b7216a8040b3a3418e588ecf2959 [term 1 FOLLOWER]: Refusing update from remote peer 599ede616cdd412fa79af64abd77799a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:34:13.476447 25495 consensus_queue.cc:1035] T 7184e4ee10824392b06d163249c4e38b P 599ede616cdd412fa79af64abd77799a [LEADER]: Connected to new peer: Peer: permanent_uuid: "14098e2fbdf1420da6d97dd9a26616f1" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 39107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:34:13.476677 25490 consensus_queue.cc:1035] T 7184e4ee10824392b06d163249c4e38b P 599ede616cdd412fa79af64abd77799a [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e9b7216a8040b3a3418e588ecf2959" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 37385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:34:13.484316 25498 mvcc.cc:205] Tried to move back new op lower bound from 7317099534227558400 to 7317099534106062848. Current Snapshot: MvccSnapshot[applied={T|T < 7317099534227558400}]
I20260810 22:34:13.490630 25501 mvcc.cc:205] Tried to move back new op lower bound from 7317099534227558400 to 7317099534106062848. Current Snapshot: MvccSnapshot[applied={T|T < 7317099534227558400}]
I20260810 22:34:13.496901 25500 mvcc.cc:205] Tried to move back new op lower bound from 7317099534227558400 to 7317099534106062848. Current Snapshot: MvccSnapshot[applied={T|T < 7317099534227558400 or (T in {7317099534227558400})}]
I20260810 22:34:13.497898 24055 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:34:13.498093 24055 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:34:13.503182 24055 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 61c3a2bed2604ebcb00d2c5647841a3c: Sending DeleteTablet for 3 replicas of tablet 7184e4ee10824392b06d163249c4e38b
I20260810 22:34:13.504019 25213 tablet_service.cc:1514] Processing DeleteTablet for tablet 7184e4ee10824392b06d163249c4e38b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:34:13 UTC) from {username='oryx', principal='oryx/127.16.233.126@KRBTEST.COM'} at 127.0.0.1:42588
I20260810 22:34:13.504578 25511 tablet_replica.cc:330] T 7184e4ee10824392b06d163249c4e38b P 83e9b7216a8040b3a3418e588ecf2959: stopping tablet replica
I20260810 22:34:13.504585 25371 tablet_service.cc:1514] Processing DeleteTablet for tablet 7184e4ee10824392b06d163249c4e38b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:34:13 UTC) from {username='oryx', principal='oryx/127.16.233.126@KRBTEST.COM'} at 127.0.0.1:55134
I20260810 22:34:13.505020 25511 raft_consensus.cc:2229] T 7184e4ee10824392b06d163249c4e38b P 83e9b7216a8040b3a3418e588ecf2959 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:34:13.505178 25512 tablet_replica.cc:330] T 7184e4ee10824392b06d163249c4e38b P 14098e2fbdf1420da6d97dd9a26616f1: stopping tablet replica
I20260810 22:34:13.505399 25511 raft_consensus.cc:2258] T 7184e4ee10824392b06d163249c4e38b P 83e9b7216a8040b3a3418e588ecf2959 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:34:13.505599 25512 raft_consensus.cc:2229] T 7184e4ee10824392b06d163249c4e38b P 14098e2fbdf1420da6d97dd9a26616f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:34:13.505954 25512 raft_consensus.cc:2258] T 7184e4ee10824392b06d163249c4e38b P 14098e2fbdf1420da6d97dd9a26616f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:34:13.506397 25062 tablet_service.cc:1514] Processing DeleteTablet for tablet 7184e4ee10824392b06d163249c4e38b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:34:13 UTC) from {username='oryx', principal='oryx/127.16.233.126@KRBTEST.COM'} at 127.0.0.1:60500
I20260810 22:34:13.508323 25511 ts_tablet_manager.cc:1905] T 7184e4ee10824392b06d163249c4e38b P 83e9b7216a8040b3a3418e588ecf2959: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:34:13.508551 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 24020
I20260810 22:34:13.511292 25512 ts_tablet_manager.cc:1905] T 7184e4ee10824392b06d163249c4e38b P 14098e2fbdf1420da6d97dd9a26616f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:34:13.514451 25512 ts_tablet_manager.cc:1918] T 7184e4ee10824392b06d163249c4e38b P 14098e2fbdf1420da6d97dd9a26616f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:34:13.514556 25511 ts_tablet_manager.cc:1918] T 7184e4ee10824392b06d163249c4e38b P 83e9b7216a8040b3a3418e588ecf2959: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:34:13.514691 25511 log.cc:1196] T 7184e4ee10824392b06d163249c4e38b P 83e9b7216a8040b3a3418e588ecf2959: Deleting WAL directory at /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-1/wal/wals/7184e4ee10824392b06d163249c4e38b
I20260810 22:34:13.514858 25512 log.cc:1196] T 7184e4ee10824392b06d163249c4e38b P 14098e2fbdf1420da6d97dd9a26616f1: Deleting WAL directory at /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-2/wal/wals/7184e4ee10824392b06d163249c4e38b
I20260810 22:34:13.516306 25512 ts_tablet_manager.cc:1939] T 7184e4ee10824392b06d163249c4e38b P 14098e2fbdf1420da6d97dd9a26616f1: Deleting consensus metadata
W20260810 22:34:13.520058 25316 connection.cc:535] server connection from 127.0.0.1:55134 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 22:34:13.515168 25511 ts_tablet_manager.cc:1939] T 7184e4ee10824392b06d163249c4e38b P 83e9b7216a8040b3a3418e588ecf2959: Deleting consensus metadata
W20260810 22:34:13.530550 25161 connection.cc:441] server connection from 127.0.0.1:42588 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42588 (request call id 1) could send its response
I20260810 22:34:13.530980 25514 tablet_replica.cc:330] T 7184e4ee10824392b06d163249c4e38b P 599ede616cdd412fa79af64abd77799a: stopping tablet replica
I20260810 22:34:13.531397 25514 raft_consensus.cc:2229] T 7184e4ee10824392b06d163249c4e38b P 599ede616cdd412fa79af64abd77799a [term 1 LEADER]: Raft consensus shutting down.
I20260810 22:34:13.531832 25459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:13.506983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50960 (local address 127.16.233.125:34439)
0810 22:34:13.507113 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:13.507117 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:13.507131 (+ 14us) server_negotiation.cc:407] Connection header received
0810 22:34:13.507171 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:13.507174 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:13.507221 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:13.507298 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:13.517867 (+ 10569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:13.518754 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:13.520312 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:13.520701 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:13.520785 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:13.526495 (+ 5710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:13.526499 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:13.526502 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:13.526536 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:13.528391 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:13.529433 (+ 1042us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:13.529437 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:13.529440 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:13.529563 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:13.529960 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:13.529963 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:13.529966 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:13.530200 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:13.530323 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:13.531521 (+ 1198us) server_negotiation.cc:299] Negotiation successful
0810 22:34:13.531656 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 22:34:13.531929 25514 raft_consensus.cc:2258] T 7184e4ee10824392b06d163249c4e38b P 599ede616cdd412fa79af64abd77799a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:34:13.534742 24863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:13.524064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50972 (local address 127.16.233.125:34439)
0810 22:34:13.524190 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:13.524195 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:13.524438 (+ 243us) server_negotiation.cc:407] Connection header received
0810 22:34:13.524610 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:13.524613 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:13.524660 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:13.524711 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:13.526242 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:13.527036 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:13.528233 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:13.528577 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:13.528645 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:13.530424 (+ 1779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:13.530427 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:13.530430 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:13.530456 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:13.532396 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:13.533195 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:13.533199 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:13.533200 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:13.533282 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:13.533538 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:13.533542 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:13.533543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:13.533748 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:13.533866 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:13.534481 (+ 615us) server_negotiation.cc:299] Negotiation successful
0810 22:34:13.534596 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 22:34:13.536552 25514 ts_tablet_manager.cc:1905] T 7184e4ee10824392b06d163249c4e38b P 599ede616cdd412fa79af64abd77799a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:34:13.540114 25514 ts_tablet_manager.cc:1918] T 7184e4ee10824392b06d163249c4e38b P 599ede616cdd412fa79af64abd77799a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:34:13.540258 25514 log.cc:1196] T 7184e4ee10824392b06d163249c4e38b P 599ede616cdd412fa79af64abd77799a: Deleting WAL directory at /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-0/wal/wals/7184e4ee10824392b06d163249c4e38b
I20260810 22:34:13.540675 25514 ts_tablet_manager.cc:1939] T 7184e4ee10824392b06d163249c4e38b P 599ede616cdd412fa79af64abd77799a: Deleting consensus metadata
W20260810 22:34:13.542053 25013 connection.cc:441] server connection from 127.0.0.1:60500 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60500 (request call id 1) could send its response
I20260810 22:34:13.543779 25140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:13.507209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52418 (local address 127.16.233.124:37989)
0810 22:34:13.507330 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:13.507336 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:13.531051 (+ 23715us) server_negotiation.cc:407] Connection header received
0810 22:34:13.531101 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:13.531104 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:13.531153 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:13.531220 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:13.533900 (+ 2680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:13.534768 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:13.537346 (+ 2578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:13.537699 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:13.537806 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:13.539628 (+ 1822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:13.539633 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:13.539636 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:13.539664 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:13.541655 (+ 1991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:13.542332 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:13.542336 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:13.542338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:13.542447 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:13.542812 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:13.542816 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:13.542818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:13.543030 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:13.543187 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:13.543456 (+ 269us) server_negotiation.cc:299] Negotiation successful
0810 22:34:13.543589 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 22:34:13.550688 25458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:13.524479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52434 (local address 127.16.233.124:37989)
0810 22:34:13.524598 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:13.524603 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:13.530316 (+ 5713us) server_negotiation.cc:407] Connection header received
0810 22:34:13.542057 (+ 11741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:13.542061 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:13.542111 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:13.542164 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:13.543153 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:13.544004 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:13.545272 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:13.545523 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:13.545610 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:13.547325 (+ 1715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:13.547328 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:13.547330 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:13.547353 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:13.548999 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:13.549494 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:13.549497 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:13.549498 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:13.549576 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:13.549858 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:13.549860 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:13.549862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:13.550045 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:13.550151 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:13.550415 (+ 264us) server_negotiation.cc:299] Negotiation successful
0810 22:34:13.550557 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 22:34:14.496048 25521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:14.495484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39177 (local address 127.16.233.67:36415)
0810 22:34:14.495825 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:14.495939 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":109,"threads_started":1}
W20260810 22:34:14.496408 25445 heartbeater.cc:644] Failed to heartbeat to 127.16.233.126:39177 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.233.126:39177: connect: Connection refused (error 111)
I20260810 22:34:14.496708 25523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:14.496127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39177 (local address 127.16.233.65:52133)
0810 22:34:14.496506 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:14.496604 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":71,"threads_started":1}
I20260810 22:34:14.496708 25522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:14.495823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39177 (local address 127.16.233.66:49183)
0810 22:34:14.496506 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:14.496604 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":87,"threads_started":1}
W20260810 22:34:14.497077 25135 heartbeater.cc:644] Failed to heartbeat to 127.16.233.126:39177 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.233.126:39177: connect: Connection refused (error 111)
W20260810 22:34:14.497206 25290 heartbeater.cc:644] Failed to heartbeat to 127.16.233.126:39177 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.233.126:39177: connect: Connection refused (error 111)
I20260810 22:34:15.003407 25524 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 61c3a2bed2604ebcb00d2c5647841a3c)
I20260810 22:34:15.003715 25524 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39177 } } peers { permanent_uuid: "580d354da5eb4d03a86d298eb6bc5af7" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 34439 } attrs { promote: false } } peers { permanent_uuid: "8059d406ab3242d3868f653160912ca1" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 37989 } attrs { promote: false } }
I20260810 22:34:15.003856 25525 raft_consensus.cc:482] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 61c3a2bed2604ebcb00d2c5647841a3c)
I20260810 22:34:15.004081 25525 raft_consensus.cc:504] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39177 } } peers { permanent_uuid: "580d354da5eb4d03a86d298eb6bc5af7" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 34439 } attrs { promote: false } } peers { permanent_uuid: "8059d406ab3242d3868f653160912ca1" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 37989 } attrs { promote: false } }
I20260810 22:34:15.004982 25524 leader_election.cc:288] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 61c3a2bed2604ebcb00d2c5647841a3c (127.16.233.126:39177), 580d354da5eb4d03a86d298eb6bc5af7 (127.16.233.125:34439)
I20260810 22:34:15.005684 25528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:15.005146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39177 (local address 127.0.0.1:46090)
0810 22:34:15.005468 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:15.005567 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":77,"threads_started":1}
I20260810 22:34:15.005909 25525 leader_election.cc:288] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 61c3a2bed2604ebcb00d2c5647841a3c (127.16.233.126:39177), 8059d406ab3242d3868f653160912ca1 (127.16.233.124:37989)
W20260810 22:34:15.005924 24921 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39177: connect: Connection refused (error 111)
I20260810 22:34:15.006070 25527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:15.005169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39177 (local address 127.0.0.1:46096)
0810 22:34:15.005884 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:15.005970 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":121,"threads_started":1}
W20260810 22:34:15.006223 24714 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39177: connect: Connection refused (error 111)
I20260810 22:34:15.007126 25527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:15.006908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39177 (local address 127.0.0.1:46104)
0810 22:34:15.007004 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:15.007057 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260810 22:34:15.007448 24714 leader_election.cc:334] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 61c3a2bed2604ebcb00d2c5647841a3c (127.16.233.126:39177): Network error: Client connection negotiation failed: client connection to 127.16.233.126:39177: connect: Connection refused (error 111)
I20260810 22:34:15.007541 25528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:15.007298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39177 (local address 127.0.0.1:46110)
0810 22:34:15.007427 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:15.007479 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260810 22:34:15.007647 24921 leader_election.cc:334] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 61c3a2bed2604ebcb00d2c5647841a3c (127.16.233.126:39177): Network error: Client connection negotiation failed: client connection to 127.16.233.126:39177: connect: Connection refused (error 111)
I20260810 22:34:15.009905 25526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:15.005216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.125:34439 (local address 127.0.0.1:50984)
0810 22:34:15.005492 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:15.005506 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:15.005621 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:15.006500 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:15.006503 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:15.006527 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:34:15.006676 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:15.006685 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:15.007812 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:15.007818 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:15.008632 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:15.008641 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:15.009436 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:15.009439 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:15.009555 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:15.009574 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 22:34:15.009665 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 22:34:15.009747 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":4,"spinlock_wait_cycles":2304,"thread_start_us":84,"threads_started":1}
I20260810 22:34:15.009953 25529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:15.005508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50984 (local address 127.16.233.125:34439)
0810 22:34:15.006208 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:15.006213 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:15.006228 (+ 15us) server_negotiation.cc:407] Connection header received
0810 22:34:15.006284 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:15.006289 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:15.006341 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:15.006775 (+ 434us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:34:15.006916 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:15.007708 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:15.008770 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:15.009281 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:15.009436 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:15.009491 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:15.009681 (+ 190us) server_negotiation.cc:299] Negotiation successful
0810 22:34:15.009755 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":73,"server-negotiator.queue_time_us":614,"thread_start_us":65,"threads_started":1}
I20260810 22:34:15.010453 25531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:15.005731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.124:37989 (local address 127.0.0.1:52446)
0810 22:34:15.006121 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:15.006138 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:15.006245 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:15.006707 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:15.006713 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:15.006735 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:34:15.006953 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:15.006963 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:15.008169 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:15.008174 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:15.009178 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:15.009187 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:15.010026 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:15.010031 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:15.010128 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:15.010156 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 22:34:15.010207 (+ 51us) client_negotiation.cc:240] Negotiation successful
0810 22:34:15.010288 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":58,"thread_start_us":70,"threads_started":1}
I20260810 22:34:15.010619 24738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8059d406ab3242d3868f653160912ca1" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "580d354da5eb4d03a86d298eb6bc5af7" is_pre_election: true
I20260810 22:34:15.010881 24738 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8059d406ab3242d3868f653160912ca1 in term 1.
I20260810 22:34:15.010954 25532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:15.005918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52446 (local address 127.16.233.124:37989)
0810 22:34:15.006498 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:15.006502 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:15.006512 (+ 10us) server_negotiation.cc:407] Connection header received
0810 22:34:15.006545 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:15.006547 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:15.006582 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:15.006845 (+ 263us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:34:15.007108 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:15.008040 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:15.009317 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:15.009888 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:15.010365 (+ 477us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:15.010415 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:15.010458 (+ 43us) server_negotiation.cc:299] Negotiation successful
0810 22:34:15.010819 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":88,"threads_started":1}
I20260810 22:34:15.011368 24922 leader_election.cc:302] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 580d354da5eb4d03a86d298eb6bc5af7, 8059d406ab3242d3868f653160912ca1; no voters: 61c3a2bed2604ebcb00d2c5647841a3c
I20260810 22:34:15.011530 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "580d354da5eb4d03a86d298eb6bc5af7" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8059d406ab3242d3868f653160912ca1" is_pre_election: true
I20260810 22:34:15.011660 25524 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 22:34:15.011771 24948 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 580d354da5eb4d03a86d298eb6bc5af7 in term 1.
I20260810 22:34:15.011864 25524 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 61c3a2bed2604ebcb00d2c5647841a3c)
I20260810 22:34:15.011924 25524 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1 [term 1 FOLLOWER]: Advancing to term 2
I20260810 22:34:15.012142 24716 leader_election.cc:302] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 580d354da5eb4d03a86d298eb6bc5af7, 8059d406ab3242d3868f653160912ca1; no voters: 61c3a2bed2604ebcb00d2c5647841a3c
I20260810 22:34:15.012396 25525 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 22:34:15.012458 25525 raft_consensus.cc:482] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 61c3a2bed2604ebcb00d2c5647841a3c)
I20260810 22:34:15.012507 25525 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7 [term 1 FOLLOWER]: Advancing to term 2
I20260810 22:34:15.013316 25524 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39177 } } peers { permanent_uuid: "580d354da5eb4d03a86d298eb6bc5af7" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 34439 } attrs { promote: false } } peers { permanent_uuid: "8059d406ab3242d3868f653160912ca1" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 37989 } attrs { promote: false } }
I20260810 22:34:15.013681 25525 raft_consensus.cc:504] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39177 } } peers { permanent_uuid: "580d354da5eb4d03a86d298eb6bc5af7" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 34439 } attrs { promote: false } } peers { permanent_uuid: "8059d406ab3242d3868f653160912ca1" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 37989 } attrs { promote: false } }
I20260810 22:34:15.013789 25524 leader_election.cc:288] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1 [CANDIDATE]: Term 2 election: Requested vote from peers 61c3a2bed2604ebcb00d2c5647841a3c (127.16.233.126:39177), 580d354da5eb4d03a86d298eb6bc5af7 (127.16.233.125:34439)
I20260810 22:34:15.014135 25525 leader_election.cc:288] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7 [CANDIDATE]: Term 2 election: Requested vote from peers 61c3a2bed2604ebcb00d2c5647841a3c (127.16.233.126:39177), 8059d406ab3242d3868f653160912ca1 (127.16.233.124:37989)
I20260810 22:34:15.014237 25526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:15.013928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39177 (local address 127.0.0.1:46124)
0810 22:34:15.014057 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:15.014152 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260810 22:34:15.014446 24738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8059d406ab3242d3868f653160912ca1" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "580d354da5eb4d03a86d298eb6bc5af7"
I20260810 22:34:15.014564 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "580d354da5eb4d03a86d298eb6bc5af7" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8059d406ab3242d3868f653160912ca1"
I20260810 22:34:15.014623 24738 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8059d406ab3242d3868f653160912ca1 in current term 2: Already voted for candidate 580d354da5eb4d03a86d298eb6bc5af7 in this term.
I20260810 22:34:15.014734 24948 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 580d354da5eb4d03a86d298eb6bc5af7 in current term 2: Already voted for candidate 8059d406ab3242d3868f653160912ca1 in this term.
I20260810 22:34:15.014981 25531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:15.014260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39177 (local address 127.0.0.1:46136)
0810 22:34:15.014396 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:15.014454 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260810 22:34:15.015488 25526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:15.015262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39177 (local address 127.0.0.1:46146)
0810 22:34:15.015377 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:15.015427 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260810 22:34:15.015596 24921 leader_election.cc:334] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 61c3a2bed2604ebcb00d2c5647841a3c (127.16.233.126:39177): Network error: Client connection negotiation failed: client connection to 127.16.233.126:39177: connect: Connection refused (error 111)
I20260810 22:34:15.015661 24921 leader_election.cc:302] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8059d406ab3242d3868f653160912ca1; no voters: 580d354da5eb4d03a86d298eb6bc5af7, 61c3a2bed2604ebcb00d2c5647841a3c
I20260810 22:34:15.015795 25524 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260810 22:34:15.015866 25531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:15.015557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39177 (local address 127.0.0.1:46156)
0810 22:34:15.015717 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:15.015775 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
W20260810 22:34:15.015944 24714 leader_election.cc:334] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 61c3a2bed2604ebcb00d2c5647841a3c (127.16.233.126:39177): Network error: Client connection negotiation failed: client connection to 127.16.233.126:39177: connect: Connection refused (error 111)
I20260810 22:34:15.016052 24714 leader_election.cc:302] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 580d354da5eb4d03a86d298eb6bc5af7; no voters: 61c3a2bed2604ebcb00d2c5647841a3c, 8059d406ab3242d3868f653160912ca1
I20260810 22:34:15.016189 25525 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260810 22:34:15.497556 25535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:15.497054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39177 (local address 127.16.233.67:54051)
0810 22:34:15.497357 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:15.497437 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":91,"threads_started":1}
I20260810 22:34:15.498210 25536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:15.497726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39177 (local address 127.16.233.65:46351)
0810 22:34:15.498018 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:15.498080 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":88,"threads_started":1}
I20260810 22:34:15.498317 25537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:15.497828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39177 (local address 127.16.233.66:54289)
0810 22:34:15.498100 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:15.498183 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":72,"threads_started":1}
I20260810 22:34:16.499507 25538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:16.498838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39177 (local address 127.16.233.67:37463)
0810 22:34:16.499308 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:16.499392 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":110,"threads_started":1}
I20260810 22:34:16.499632 25540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:16.499165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39177 (local address 127.16.233.65:43407)
0810 22:34:16.499485 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:16.499541 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":81,"threads_started":1}
I20260810 22:34:16.499571 25539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:16.499109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39177 (local address 127.16.233.66:40719)
0810 22:34:16.499400 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:16.499465 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":134,"threads_started":1}
I20260810 22:34:16.812392 25541 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:34:16.812556 25541 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39177 } } peers { permanent_uuid: "580d354da5eb4d03a86d298eb6bc5af7" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 34439 } attrs { promote: false } } peers { permanent_uuid: "8059d406ab3242d3868f653160912ca1" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 37989 } attrs { promote: false } }
I20260810 22:34:16.813246 25541 leader_election.cc:288] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 61c3a2bed2604ebcb00d2c5647841a3c (127.16.233.126:39177), 580d354da5eb4d03a86d298eb6bc5af7 (127.16.233.125:34439)
I20260810 22:34:16.813825 24738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8059d406ab3242d3868f653160912ca1" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "580d354da5eb4d03a86d298eb6bc5af7" is_pre_election: true
I20260810 22:34:16.814052 24738 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8059d406ab3242d3868f653160912ca1 in term 2.
I20260810 22:34:16.814092 25542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:16.813508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39177 (local address 127.0.0.1:46386)
0810 22:34:16.813898 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:16.813966 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":95,"threads_started":1}
I20260810 22:34:16.814455 24922 leader_election.cc:302] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 580d354da5eb4d03a86d298eb6bc5af7, 8059d406ab3242d3868f653160912ca1; no voters:
I20260810 22:34:16.814656 25541 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 22:34:16.814725 25541 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:34:16.814764 25541 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1 [term 2 FOLLOWER]: Advancing to term 3
I20260810 22:34:16.815443 25542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:16.815095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39177 (local address 127.0.0.1:46402)
0810 22:34:16.815276 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:16.815345 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260810 22:34:16.815573 24921 leader_election.cc:334] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 61c3a2bed2604ebcb00d2c5647841a3c (127.16.233.126:39177): Network error: Client connection negotiation failed: client connection to 127.16.233.126:39177: connect: Connection refused (error 111)
I20260810 22:34:16.816133 25541 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39177 } } peers { permanent_uuid: "580d354da5eb4d03a86d298eb6bc5af7" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 34439 } attrs { promote: false } } peers { permanent_uuid: "8059d406ab3242d3868f653160912ca1" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 37989 } attrs { promote: false } }
I20260810 22:34:16.816603 25541 leader_election.cc:288] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1 [CANDIDATE]: Term 3 election: Requested vote from peers 61c3a2bed2604ebcb00d2c5647841a3c (127.16.233.126:39177), 580d354da5eb4d03a86d298eb6bc5af7 (127.16.233.125:34439)
I20260810 22:34:16.817081 24738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8059d406ab3242d3868f653160912ca1" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "580d354da5eb4d03a86d298eb6bc5af7"
I20260810 22:34:16.817261 24738 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7 [term 2 FOLLOWER]: Advancing to term 3
I20260810 22:34:16.817307 25542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:16.816989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39177 (local address 127.0.0.1:46412)
0810 22:34:16.817116 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:16.817171 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260810 22:34:16.818408 25542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:16.818054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39177 (local address 127.0.0.1:46426)
0810 22:34:16.818199 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:16.818259 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260810 22:34:16.818584 24921 leader_election.cc:334] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 61c3a2bed2604ebcb00d2c5647841a3c (127.16.233.126:39177): Network error: Client connection negotiation failed: client connection to 127.16.233.126:39177: connect: Connection refused (error 111)
I20260810 22:34:16.818759 24738 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8059d406ab3242d3868f653160912ca1 in term 3.
I20260810 22:34:16.819216 24922 leader_election.cc:302] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 580d354da5eb4d03a86d298eb6bc5af7, 8059d406ab3242d3868f653160912ca1; no voters: 61c3a2bed2604ebcb00d2c5647841a3c
I20260810 22:34:16.819538 25541 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1 [term 3 FOLLOWER]: Leader election won for term 3
I20260810 22:34:16.819793 25541 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1 [term 3 LEADER]: Becoming Leader. State: Replica: 8059d406ab3242d3868f653160912ca1, State: Running, Role: LEADER
I20260810 22:34:16.820122 25541 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39177 } } peers { permanent_uuid: "580d354da5eb4d03a86d298eb6bc5af7" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 34439 } attrs { promote: false } } peers { permanent_uuid: "8059d406ab3242d3868f653160912ca1" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 37989 } attrs { promote: false } }
I20260810 22:34:16.821467 25544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8059d406ab3242d3868f653160912ca1. Latest consensus state: current_term: 3 leader_uuid: "8059d406ab3242d3868f653160912ca1" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39177 } } peers { permanent_uuid: "580d354da5eb4d03a86d298eb6bc5af7" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 34439 } attrs { promote: false } } peers { permanent_uuid: "8059d406ab3242d3868f653160912ca1" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 37989 } attrs { promote: false } } }
I20260810 22:34:16.821633 25544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1 [sys.catalog]: This master's current role is: LEADER
I20260810 22:34:16.822144 25546 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:34:16.825253 25546 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:34:16.825871 25546 catalog_manager.cc:1259] Loaded cluster ID: 0d14ee0609a54fd5bfab4629910fd236
I20260810 22:34:16.825938 25546 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:34:16.826674 25546 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:34:16.827201 25546 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1: Loaded TSK: 0
I20260810 22:34:16.828058 25546 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:34:16.863197 24936 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52434:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 22:34:16.866603 24936 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:34:16.870487 24738 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7 [term 3 FOLLOWER]: Refusing update from remote peer 8059d406ab3242d3868f653160912ca1: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260810 22:34:16.870620 25542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:16.870278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39177 (local address 127.0.0.1:46434)
0810 22:34:16.870440 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:16.870506 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260810 22:34:16.871052 25544 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "580d354da5eb4d03a86d298eb6bc5af7" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 34439 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260810 22:34:16.871582 25542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:16.871307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39177 (local address 127.0.0.1:46438)
0810 22:34:16.871437 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:16.871511 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260810 22:34:16.871716 24921 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1 -> Peer 61c3a2bed2604ebcb00d2c5647841a3c (127.16.233.126:39177): Couldn't send request to peer 61c3a2bed2604ebcb00d2c5647841a3c. Status: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39177: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 22:34:16.872684 25548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8059d406ab3242d3868f653160912ca1. Latest consensus state: current_term: 3 leader_uuid: "8059d406ab3242d3868f653160912ca1" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39177 } } peers { permanent_uuid: "580d354da5eb4d03a86d298eb6bc5af7" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 34439 } attrs { promote: false } } peers { permanent_uuid: "8059d406ab3242d3868f653160912ca1" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 37989 } attrs { promote: false } } }
I20260810 22:34:16.872864 25548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 22:34:16.874151 25541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "8059d406ab3242d3868f653160912ca1" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39177 } } peers { permanent_uuid: "580d354da5eb4d03a86d298eb6bc5af7" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 34439 } attrs { promote: false } } peers { permanent_uuid: "8059d406ab3242d3868f653160912ca1" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 37989 } attrs { promote: false } } }
I20260810 22:34:16.874289 25541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1 [sys.catalog]: This master's current role is: LEADER
I20260810 22:34:16.875068 25548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "8059d406ab3242d3868f653160912ca1" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "61c3a2bed2604ebcb00d2c5647841a3c" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39177 } } peers { permanent_uuid: "580d354da5eb4d03a86d298eb6bc5af7" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 34439 } attrs { promote: false } } peers { permanent_uuid: "8059d406ab3242d3868f653160912ca1" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 37989 } attrs { promote: false } } }
I20260810 22:34:16.875281 25548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 580d354da5eb4d03a86d298eb6bc5af7 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 22:34:16.886792 25555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:16.882317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55136 (local address 127.16.233.67:39107)
0810 22:34:16.882573 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:16.882577 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:16.882905 (+ 328us) server_negotiation.cc:407] Connection header received
0810 22:34:16.883122 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:16.883125 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:16.883178 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:16.883269 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:34:16.883610 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:16.884202 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:16.885489 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:16.886134 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:16.886263 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:16.886332 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:16.886543 (+ 211us) server_negotiation.cc:299] Negotiation successful
0810 22:34:16.886631 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":106,"threads_started":1}
I20260810 22:34:16.886962 25542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:16.881547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.65:39951 (local address 127.0.0.1:60520)
0810 22:34:16.881681 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:16.881699 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:16.881806 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:16.882543 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:16.882546 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:16.882573 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:34:16.882714 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:16.882722 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:16.884219 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:16.884223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:16.885528 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:16.885537 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:16.886492 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:16.886495 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:16.886627 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:16.886662 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 22:34:16.886712 (+ 50us) client_negotiation.cc:240] Negotiation successful
0810 22:34:16.886793 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":35}
I20260810 22:34:16.886792 25556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:16.882206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.67:39107 (local address 127.0.0.1:55136)
0810 22:34:16.882812 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:16.882824 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:16.883010 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:16.883385 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:16.883388 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:16.883413 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:34:16.883496 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:16.883501 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:16.884298 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:16.884301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:16.885329 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:16.885343 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:16.886281 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:16.886285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:16.886378 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:16.886402 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 22:34:16.886526 (+ 124us) client_negotiation.cc:240] Negotiation successful
0810 22:34:16.886632 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"mutex_wait_us":1,"thread_start_us":128,"threads_started":1}
I20260810 22:34:16.887210 25553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:16.881831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60520 (local address 127.16.233.65:39951)
0810 22:34:16.882259 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:16.882265 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:16.882283 (+ 18us) server_negotiation.cc:407] Connection header received
0810 22:34:16.882346 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:16.882351 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:16.882405 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:16.883178 (+ 773us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:34:16.883325 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:16.884086 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:16.885668 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:16.886316 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:16.886884 (+ 568us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:16.886954 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:16.886992 (+ 38us) server_negotiation.cc:299] Negotiation successful
0810 22:34:16.887068 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":125,"threads_started":1}
I20260810 22:34:16.887827 25062 tablet_service.cc:1467] Processing CreateTablet for tablet c9d829756d074e1fa70ae429bb52dae3 (DEFAULT_TABLE table=test-table [id=8fd6504594d349dc89ebdacf159097ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:34:16.887915 25554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:16.882582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.66:37385 (local address 127.0.0.1:42596)
0810 22:34:16.883575 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:16.883590 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:16.883687 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:16.884140 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:16.884143 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:16.884162 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:34:16.884289 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:16.884296 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:16.885365 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:16.885369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:16.886573 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:16.886582 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:16.887453 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:16.887457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:16.887567 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:16.887589 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 22:34:16.887688 (+ 99us) client_negotiation.cc:240] Negotiation successful
0810 22:34:16.887764 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":929,"thread_start_us":103,"threads_started":1}
I20260810 22:34:16.887928 25557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:16.882971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42596 (local address 127.16.233.66:37385)
0810 22:34:16.883901 (+ 930us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:16.883906 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:16.883918 (+ 12us) server_negotiation.cc:407] Connection header received
0810 22:34:16.883966 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:16.883968 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:16.884010 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:16.884112 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:34:16.884397 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:16.885223 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:16.886723 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:16.887339 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:16.887450 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:16.887510 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:16.887708 (+ 198us) server_negotiation.cc:299] Negotiation successful
0810 22:34:16.887788 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":847,"thread_start_us":95,"threads_started":1}
I20260810 22:34:16.888208 25062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9d829756d074e1fa70ae429bb52dae3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:34:16.889456 25213 tablet_service.cc:1467] Processing CreateTablet for tablet c9d829756d074e1fa70ae429bb52dae3 (DEFAULT_TABLE table=test-table [id=8fd6504594d349dc89ebdacf159097ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:34:16.889782 25213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9d829756d074e1fa70ae429bb52dae3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:34:16.890954 25558 tablet_bootstrap.cc:492] T c9d829756d074e1fa70ae429bb52dae3 P 599ede616cdd412fa79af64abd77799a: Bootstrap starting.
I20260810 22:34:16.891188 25371 tablet_service.cc:1467] Processing CreateTablet for tablet c9d829756d074e1fa70ae429bb52dae3 (DEFAULT_TABLE table=test-table [id=8fd6504594d349dc89ebdacf159097ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:34:16.891506 25371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9d829756d074e1fa70ae429bb52dae3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:34:16.892076 25558 tablet_bootstrap.cc:654] T c9d829756d074e1fa70ae429bb52dae3 P 599ede616cdd412fa79af64abd77799a: Neither blocks nor log segments found. Creating new log.
I20260810 22:34:16.892119 25559 tablet_bootstrap.cc:492] T c9d829756d074e1fa70ae429bb52dae3 P 83e9b7216a8040b3a3418e588ecf2959: Bootstrap starting.
I20260810 22:34:16.893180 25559 tablet_bootstrap.cc:654] T c9d829756d074e1fa70ae429bb52dae3 P 83e9b7216a8040b3a3418e588ecf2959: Neither blocks nor log segments found. Creating new log.
I20260810 22:34:16.893793 25558 tablet_bootstrap.cc:492] T c9d829756d074e1fa70ae429bb52dae3 P 599ede616cdd412fa79af64abd77799a: No bootstrap required, opened a new log
I20260810 22:34:16.893939 25558 ts_tablet_manager.cc:1397] T c9d829756d074e1fa70ae429bb52dae3 P 599ede616cdd412fa79af64abd77799a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 22:34:16.894345 25558 raft_consensus.cc:348] T c9d829756d074e1fa70ae429bb52dae3 P 599ede616cdd412fa79af64abd77799a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599ede616cdd412fa79af64abd77799a" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 39951 } } peers { permanent_uuid: "83e9b7216a8040b3a3418e588ecf2959" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 37385 } } peers { permanent_uuid: "14098e2fbdf1420da6d97dd9a26616f1" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 39107 } }
I20260810 22:34:16.894457 25562 tablet_bootstrap.cc:492] T c9d829756d074e1fa70ae429bb52dae3 P 14098e2fbdf1420da6d97dd9a26616f1: Bootstrap starting.
I20260810 22:34:16.894634 25558 raft_consensus.cc:374] T c9d829756d074e1fa70ae429bb52dae3 P 599ede616cdd412fa79af64abd77799a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:34:16.894740 25558 raft_consensus.cc:729] T c9d829756d074e1fa70ae429bb52dae3 P 599ede616cdd412fa79af64abd77799a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 599ede616cdd412fa79af64abd77799a, State: Initialized, Role: FOLLOWER
I20260810 22:34:16.894943 25558 consensus_queue.cc:260] T c9d829756d074e1fa70ae429bb52dae3 P 599ede616cdd412fa79af64abd77799a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599ede616cdd412fa79af64abd77799a" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 39951 } } peers { permanent_uuid: "83e9b7216a8040b3a3418e588ecf2959" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 37385 } } peers { permanent_uuid: "14098e2fbdf1420da6d97dd9a26616f1" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 39107 } }
I20260810 22:34:16.895248 25559 tablet_bootstrap.cc:492] T c9d829756d074e1fa70ae429bb52dae3 P 83e9b7216a8040b3a3418e588ecf2959: No bootstrap required, opened a new log
I20260810 22:34:16.895370 25558 ts_tablet_manager.cc:1428] T c9d829756d074e1fa70ae429bb52dae3 P 599ede616cdd412fa79af64abd77799a: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260810 22:34:16.895385 25559 ts_tablet_manager.cc:1397] T c9d829756d074e1fa70ae429bb52dae3 P 83e9b7216a8040b3a3418e588ecf2959: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 22:34:16.895634 25562 tablet_bootstrap.cc:654] T c9d829756d074e1fa70ae429bb52dae3 P 14098e2fbdf1420da6d97dd9a26616f1: Neither blocks nor log segments found. Creating new log.
I20260810 22:34:16.895892 25559 raft_consensus.cc:348] T c9d829756d074e1fa70ae429bb52dae3 P 83e9b7216a8040b3a3418e588ecf2959 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599ede616cdd412fa79af64abd77799a" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 39951 } } peers { permanent_uuid: "83e9b7216a8040b3a3418e588ecf2959" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 37385 } } peers { permanent_uuid: "14098e2fbdf1420da6d97dd9a26616f1" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 39107 } }
I20260810 22:34:16.896054 25559 raft_consensus.cc:374] T c9d829756d074e1fa70ae429bb52dae3 P 83e9b7216a8040b3a3418e588ecf2959 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:34:16.896118 25559 raft_consensus.cc:729] T c9d829756d074e1fa70ae429bb52dae3 P 83e9b7216a8040b3a3418e588ecf2959 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e9b7216a8040b3a3418e588ecf2959, State: Initialized, Role: FOLLOWER
I20260810 22:34:16.896272 25559 consensus_queue.cc:260] T c9d829756d074e1fa70ae429bb52dae3 P 83e9b7216a8040b3a3418e588ecf2959 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599ede616cdd412fa79af64abd77799a" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 39951 } } peers { permanent_uuid: "83e9b7216a8040b3a3418e588ecf2959" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 37385 } } peers { permanent_uuid: "14098e2fbdf1420da6d97dd9a26616f1" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 39107 } }
I20260810 22:34:16.896334 25540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:16.895906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39177 (local address 127.16.233.65:43667)
0810 22:34:16.896185 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:16.896254 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178}
I20260810 22:34:16.896688 25559 ts_tablet_manager.cc:1428] T c9d829756d074e1fa70ae429bb52dae3 P 83e9b7216a8040b3a3418e588ecf2959: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 22:34:16.898836 25289 heartbeater.cc:503] Master 127.16.233.124:37989 was elected leader, sending a full tablet report...
I20260810 22:34:16.898941 25539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:16.898004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39177 (local address 127.16.233.66:34497)
0810 22:34:16.898789 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:16.898854 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":688}
I20260810 22:34:16.899160 25540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:16.898878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39177 (local address 127.16.233.65:32917)
0810 22:34:16.899041 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:16.899093 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260810 22:34:16.901114 25134 heartbeater.cc:503] Master 127.16.233.124:37989 was elected leader, sending a full tablet report...
I20260810 22:34:16.903749 25562 tablet_bootstrap.cc:492] T c9d829756d074e1fa70ae429bb52dae3 P 14098e2fbdf1420da6d97dd9a26616f1: No bootstrap required, opened a new log
I20260810 22:34:16.903875 25562 ts_tablet_manager.cc:1397] T c9d829756d074e1fa70ae429bb52dae3 P 14098e2fbdf1420da6d97dd9a26616f1: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260810 22:34:16.904448 25562 raft_consensus.cc:348] T c9d829756d074e1fa70ae429bb52dae3 P 14098e2fbdf1420da6d97dd9a26616f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599ede616cdd412fa79af64abd77799a" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 39951 } } peers { permanent_uuid: "83e9b7216a8040b3a3418e588ecf2959" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 37385 } } peers { permanent_uuid: "14098e2fbdf1420da6d97dd9a26616f1" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 39107 } }
I20260810 22:34:16.904592 25562 raft_consensus.cc:374] T c9d829756d074e1fa70ae429bb52dae3 P 14098e2fbdf1420da6d97dd9a26616f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:34:16.904637 25562 raft_consensus.cc:729] T c9d829756d074e1fa70ae429bb52dae3 P 14098e2fbdf1420da6d97dd9a26616f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14098e2fbdf1420da6d97dd9a26616f1, State: Initialized, Role: FOLLOWER
I20260810 22:34:16.904778 25562 consensus_queue.cc:260] T c9d829756d074e1fa70ae429bb52dae3 P 14098e2fbdf1420da6d97dd9a26616f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599ede616cdd412fa79af64abd77799a" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 39951 } } peers { permanent_uuid: "83e9b7216a8040b3a3418e588ecf2959" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 37385 } } peers { permanent_uuid: "14098e2fbdf1420da6d97dd9a26616f1" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 39107 } }
I20260810 22:34:16.905210 25562 ts_tablet_manager.cc:1428] T c9d829756d074e1fa70ae429bb52dae3 P 14098e2fbdf1420da6d97dd9a26616f1: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260810 22:34:16.905941 25538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:16.905655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39177 (local address 127.16.233.67:58359)
0810 22:34:16.905787 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:16.905854 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260810 22:34:16.906911 25444 heartbeater.cc:503] Master 127.16.233.124:37989 was elected leader, sending a full tablet report...
I20260810 22:34:16.907300 25538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:16.907005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39177 (local address 127.16.233.67:42389)
0810 22:34:16.907157 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:16.907217 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260810 22:34:17.076570 25566 raft_consensus.cc:482] T c9d829756d074e1fa70ae429bb52dae3 P 14098e2fbdf1420da6d97dd9a26616f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:34:17.077021 25566 raft_consensus.cc:504] T c9d829756d074e1fa70ae429bb52dae3 P 14098e2fbdf1420da6d97dd9a26616f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599ede616cdd412fa79af64abd77799a" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 39951 } } peers { permanent_uuid: "83e9b7216a8040b3a3418e588ecf2959" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 37385 } } peers { permanent_uuid: "14098e2fbdf1420da6d97dd9a26616f1" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 39107 } }
I20260810 22:34:17.078596 25566 leader_election.cc:288] T c9d829756d074e1fa70ae429bb52dae3 P 14098e2fbdf1420da6d97dd9a26616f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 599ede616cdd412fa79af64abd77799a (127.16.233.65:39951), 83e9b7216a8040b3a3418e588ecf2959 (127.16.233.66:37385)
I20260810 22:34:17.084250 25553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:17.079081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:53947 (local address 127.16.233.65:39951)
0810 22:34:17.079259 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:17.079264 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:17.079466 (+ 202us) server_negotiation.cc:407] Connection header received
0810 22:34:17.079629 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:17.079631 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:17.079675 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:17.079748 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:34:17.080107 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:17.080952 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:17.082652 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:17.083300 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:17.083441 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:17.083497 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:17.083978 (+ 481us) server_negotiation.cc:299] Negotiation successful
0810 22:34:17.084078 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 22:34:17.084250 25557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:17.079319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:56223 (local address 127.16.233.66:37385)
0810 22:34:17.079470 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:17.079475 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:17.079915 (+ 440us) server_negotiation.cc:407] Connection header received
0810 22:34:17.080007 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:17.080009 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:17.080067 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:17.080153 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:34:17.080463 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:17.081189 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:17.082648 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:17.083297 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:17.083441 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:17.083497 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:17.083978 (+ 481us) server_negotiation.cc:299] Negotiation successful
0810 22:34:17.084078 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 22:34:17.084250 25538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:17.078784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.65:39951 (local address 127.16.233.67:53947)
0810 22:34:17.079350 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:17.079367 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:17.079523 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:17.079794 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:17.079798 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:17.079820 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:34:17.079989 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:17.079997 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:17.081064 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:17.081066 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:17.082472 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:17.082486 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:17.083471 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:17.083475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:17.083704 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:17.083808 (+ 104us) client_negotiation.cc:769] Sending connection context
0810 22:34:17.083899 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 22:34:17.084064 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"mutex_wait_us":135}
I20260810 22:34:17.084263 25567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:17.078881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.66:37385 (local address 127.16.233.67:56223)
0810 22:34:17.079803 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:17.079816 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:17.079921 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:17.080219 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:17.080222 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:17.080246 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:34:17.080345 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:17.080350 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:17.081299 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:17.081302 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:17.082471 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:17.082486 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:17.083471 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:17.083475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:17.083701 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:17.083808 (+ 107us) client_negotiation.cc:769] Sending connection context
0810 22:34:17.083899 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 22:34:17.084064 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":828,"mutex_wait_us":127,"thread_start_us":118,"threads_started":1}
I20260810 22:34:17.084903 25235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9d829756d074e1fa70ae429bb52dae3" candidate_uuid: "14098e2fbdf1420da6d97dd9a26616f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e9b7216a8040b3a3418e588ecf2959" is_pre_election: true
I20260810 22:34:17.085103 25235 raft_consensus.cc:2454] T c9d829756d074e1fa70ae429bb52dae3 P 83e9b7216a8040b3a3418e588ecf2959 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14098e2fbdf1420da6d97dd9a26616f1 in term 0.
I20260810 22:34:17.085376 25084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9d829756d074e1fa70ae429bb52dae3" candidate_uuid: "14098e2fbdf1420da6d97dd9a26616f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "599ede616cdd412fa79af64abd77799a" is_pre_election: true
I20260810 22:34:17.085554 25318 leader_election.cc:302] T c9d829756d074e1fa70ae429bb52dae3 P 14098e2fbdf1420da6d97dd9a26616f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14098e2fbdf1420da6d97dd9a26616f1, 83e9b7216a8040b3a3418e588ecf2959; no voters:
I20260810 22:34:17.085646 25084 raft_consensus.cc:2454] T c9d829756d074e1fa70ae429bb52dae3 P 599ede616cdd412fa79af64abd77799a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14098e2fbdf1420da6d97dd9a26616f1 in term 0.
I20260810 22:34:17.085855 25566 raft_consensus.cc:2781] T c9d829756d074e1fa70ae429bb52dae3 P 14098e2fbdf1420da6d97dd9a26616f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:34:17.085938 25566 raft_consensus.cc:482] T c9d829756d074e1fa70ae429bb52dae3 P 14098e2fbdf1420da6d97dd9a26616f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:34:17.086037 25566 raft_consensus.cc:3037] T c9d829756d074e1fa70ae429bb52dae3 P 14098e2fbdf1420da6d97dd9a26616f1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:34:17.089184 25566 raft_consensus.cc:504] T c9d829756d074e1fa70ae429bb52dae3 P 14098e2fbdf1420da6d97dd9a26616f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599ede616cdd412fa79af64abd77799a" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 39951 } } peers { permanent_uuid: "83e9b7216a8040b3a3418e588ecf2959" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 37385 } } peers { permanent_uuid: "14098e2fbdf1420da6d97dd9a26616f1" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 39107 } }
I20260810 22:34:17.089725 25566 leader_election.cc:288] T c9d829756d074e1fa70ae429bb52dae3 P 14098e2fbdf1420da6d97dd9a26616f1 [CANDIDATE]: Term 1 election: Requested vote from peers 599ede616cdd412fa79af64abd77799a (127.16.233.65:39951), 83e9b7216a8040b3a3418e588ecf2959 (127.16.233.66:37385)
I20260810 22:34:17.090348 25235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9d829756d074e1fa70ae429bb52dae3" candidate_uuid: "14098e2fbdf1420da6d97dd9a26616f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83e9b7216a8040b3a3418e588ecf2959"
I20260810 22:34:17.090497 25235 raft_consensus.cc:3037] T c9d829756d074e1fa70ae429bb52dae3 P 83e9b7216a8040b3a3418e588ecf2959 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:34:17.091637 25235 raft_consensus.cc:2454] T c9d829756d074e1fa70ae429bb52dae3 P 83e9b7216a8040b3a3418e588ecf2959 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14098e2fbdf1420da6d97dd9a26616f1 in term 1.
I20260810 22:34:17.091987 25084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9d829756d074e1fa70ae429bb52dae3" candidate_uuid: "14098e2fbdf1420da6d97dd9a26616f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "599ede616cdd412fa79af64abd77799a"
I20260810 22:34:17.092042 25318 leader_election.cc:302] T c9d829756d074e1fa70ae429bb52dae3 P 14098e2fbdf1420da6d97dd9a26616f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14098e2fbdf1420da6d97dd9a26616f1, 83e9b7216a8040b3a3418e588ecf2959; no voters:
I20260810 22:34:17.092168 25084 raft_consensus.cc:3037] T c9d829756d074e1fa70ae429bb52dae3 P 599ede616cdd412fa79af64abd77799a [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:34:17.092365 25566 raft_consensus.cc:2781] T c9d829756d074e1fa70ae429bb52dae3 P 14098e2fbdf1420da6d97dd9a26616f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:34:17.092643 25566 raft_consensus.cc:686] T c9d829756d074e1fa70ae429bb52dae3 P 14098e2fbdf1420da6d97dd9a26616f1 [term 1 LEADER]: Becoming Leader. State: Replica: 14098e2fbdf1420da6d97dd9a26616f1, State: Running, Role: LEADER
I20260810 22:34:17.093077 25566 consensus_queue.cc:237] T c9d829756d074e1fa70ae429bb52dae3 P 14098e2fbdf1420da6d97dd9a26616f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599ede616cdd412fa79af64abd77799a" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 39951 } } peers { permanent_uuid: "83e9b7216a8040b3a3418e588ecf2959" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 37385 } } peers { permanent_uuid: "14098e2fbdf1420da6d97dd9a26616f1" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 39107 } }
I20260810 22:34:17.093641 25567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:17.093329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39177 (local address 127.16.233.67:46709)
0810 22:34:17.093478 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:17.093551 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260810 22:34:17.094183 25084 raft_consensus.cc:2454] T c9d829756d074e1fa70ae429bb52dae3 P 599ede616cdd412fa79af64abd77799a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14098e2fbdf1420da6d97dd9a26616f1 in term 1.
I20260810 22:34:17.096094 24936 catalog_manager.cc:5495] T c9d829756d074e1fa70ae429bb52dae3 P 14098e2fbdf1420da6d97dd9a26616f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 14098e2fbdf1420da6d97dd9a26616f1 (127.16.233.67). New cstate: current_term: 1 leader_uuid: "14098e2fbdf1420da6d97dd9a26616f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599ede616cdd412fa79af64abd77799a" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 39951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83e9b7216a8040b3a3418e588ecf2959" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 37385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14098e2fbdf1420da6d97dd9a26616f1" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 39107 } health_report { overall_health: HEALTHY } } }
I20260810 22:34:17.132180 25555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:17.127225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55148 (local address 127.16.233.67:39107)
0810 22:34:17.127369 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:17.127372 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:17.127442 (+ 70us) server_negotiation.cc:407] Connection header received
0810 22:34:17.127655 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:17.127658 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:17.127718 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:17.127894 (+ 176us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:34:17.128491 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:17.129229 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:17.130395 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:17.130623 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:17.130715 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:17.131099 (+ 384us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:34:17.131594 (+ 495us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:34:17.131699 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:17.131859 (+ 160us) server_negotiation.cc:299] Negotiation successful
0810 22:34:17.131968 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 22:34:17.135406 25235 raft_consensus.cc:1261] T c9d829756d074e1fa70ae429bb52dae3 P 83e9b7216a8040b3a3418e588ecf2959 [term 1 FOLLOWER]: Refusing update from remote peer 14098e2fbdf1420da6d97dd9a26616f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:34:17.135578 25084 raft_consensus.cc:1261] T c9d829756d074e1fa70ae429bb52dae3 P 599ede616cdd412fa79af64abd77799a [term 1 FOLLOWER]: Refusing update from remote peer 14098e2fbdf1420da6d97dd9a26616f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:34:17.136054 25568 consensus_queue.cc:1035] T c9d829756d074e1fa70ae429bb52dae3 P 14098e2fbdf1420da6d97dd9a26616f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e9b7216a8040b3a3418e588ecf2959" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 37385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:34:17.136310 25566 consensus_queue.cc:1035] T c9d829756d074e1fa70ae429bb52dae3 P 14098e2fbdf1420da6d97dd9a26616f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "599ede616cdd412fa79af64abd77799a" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 39951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:34:17.137943 25539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:17.137616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39177 (local address 127.16.233.66:51015)
0810 22:34:17.137788 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:17.137854 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260810 22:34:17.140106 25540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:17.139408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39177 (local address 127.16.233.65:55119)
0810 22:34:17.139939 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:17.140012 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436}
I20260810 22:34:17.141069 25567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:17.140771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39177 (local address 127.16.233.67:35465)
0810 22:34:17.140922 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:17.140981 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260810 22:34:17.141507 25539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:17.141220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39177 (local address 127.16.233.66:57667)
0810 22:34:17.141385 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:17.141436 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260810 22:34:17.146229 25567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:17.145887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39177 (local address 127.16.233.67:38467)
0810 22:34:17.146056 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:17.146130 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260810 22:34:17.146961 25540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:17.146688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39177 (local address 127.16.233.65:37075)
0810 22:34:17.146823 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:17.146884 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260810 22:34:17.152305 24936 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52434:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:34:17.152514 24936 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52434:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:34:17.157459 24936 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8059d406ab3242d3868f653160912ca1: Sending DeleteTablet for 3 replicas of tablet c9d829756d074e1fa70ae429bb52dae3
I20260810 22:34:17.158200 25062 tablet_service.cc:1514] Processing DeleteTablet for tablet c9d829756d074e1fa70ae429bb52dae3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:34:17 UTC) from {username='oryx', principal='oryx/127.16.233.124@KRBTEST.COM'} at 127.0.0.1:60520
I20260810 22:34:17.158677 25582 tablet_replica.cc:330] T c9d829756d074e1fa70ae429bb52dae3 P 599ede616cdd412fa79af64abd77799a: stopping tablet replica
I20260810 22:34:17.158855 25582 raft_consensus.cc:2229] T c9d829756d074e1fa70ae429bb52dae3 P 599ede616cdd412fa79af64abd77799a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:34:17.159029 25582 raft_consensus.cc:2258] T c9d829756d074e1fa70ae429bb52dae3 P 599ede616cdd412fa79af64abd77799a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:34:17.159238 25213 tablet_service.cc:1514] Processing DeleteTablet for tablet c9d829756d074e1fa70ae429bb52dae3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:34:17 UTC) from {username='oryx', principal='oryx/127.16.233.124@KRBTEST.COM'} at 127.0.0.1:42596
I20260810 22:34:17.159675 25583 tablet_replica.cc:330] T c9d829756d074e1fa70ae429bb52dae3 P 83e9b7216a8040b3a3418e588ecf2959: stopping tablet replica
I20260810 22:34:17.159852 25583 raft_consensus.cc:2229] T c9d829756d074e1fa70ae429bb52dae3 P 83e9b7216a8040b3a3418e588ecf2959 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:34:17.160049 25583 raft_consensus.cc:2258] T c9d829756d074e1fa70ae429bb52dae3 P 83e9b7216a8040b3a3418e588ecf2959 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:34:17.160226 25371 tablet_service.cc:1514] Processing DeleteTablet for tablet c9d829756d074e1fa70ae429bb52dae3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:34:17 UTC) from {username='oryx', principal='oryx/127.16.233.124@KRBTEST.COM'} at 127.0.0.1:55136
I20260810 22:34:17.160373 25582 ts_tablet_manager.cc:1905] T c9d829756d074e1fa70ae429bb52dae3 P 599ede616cdd412fa79af64abd77799a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:34:17.160696 25584 tablet_replica.cc:330] T c9d829756d074e1fa70ae429bb52dae3 P 14098e2fbdf1420da6d97dd9a26616f1: stopping tablet replica
I20260810 22:34:17.160915 25584 raft_consensus.cc:2229] T c9d829756d074e1fa70ae429bb52dae3 P 14098e2fbdf1420da6d97dd9a26616f1 [term 1 LEADER]: Raft consensus shutting down.
I20260810 22:34:17.161337 25584 raft_consensus.cc:2258] T c9d829756d074e1fa70ae429bb52dae3 P 14098e2fbdf1420da6d97dd9a26616f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:34:17.161526 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 24886
I20260810 22:34:17.163905 25584 ts_tablet_manager.cc:1905] T c9d829756d074e1fa70ae429bb52dae3 P 14098e2fbdf1420da6d97dd9a26616f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:34:17.167867 25584 ts_tablet_manager.cc:1918] T c9d829756d074e1fa70ae429bb52dae3 P 14098e2fbdf1420da6d97dd9a26616f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:34:17.167999 25584 log.cc:1196] T c9d829756d074e1fa70ae429bb52dae3 P 14098e2fbdf1420da6d97dd9a26616f1: Deleting WAL directory at /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-2/wal/wals/c9d829756d074e1fa70ae429bb52dae3
I20260810 22:34:17.168599 25584 ts_tablet_manager.cc:1939] T c9d829756d074e1fa70ae429bb52dae3 P 14098e2fbdf1420da6d97dd9a26616f1: Deleting consensus metadata
I20260810 22:34:17.169718 25583 ts_tablet_manager.cc:1905] T c9d829756d074e1fa70ae429bb52dae3 P 83e9b7216a8040b3a3418e588ecf2959: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:34:17.173687 24920 catalog_manager.cc:4841] TS 14098e2fbdf1420da6d97dd9a26616f1 (127.16.233.67:39107): tablet c9d829756d074e1fa70ae429bb52dae3 (table test-table [id=8fd6504594d349dc89ebdacf159097ac]) successfully deleted
I20260810 22:34:17.173744 25583 ts_tablet_manager.cc:1918] T c9d829756d074e1fa70ae429bb52dae3 P 83e9b7216a8040b3a3418e588ecf2959: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:34:17.173851 25583 log.cc:1196] T c9d829756d074e1fa70ae429bb52dae3 P 83e9b7216a8040b3a3418e588ecf2959: Deleting WAL directory at /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786401215402979-17317-0/minicluster-data/ts-1/wal/wals/c9d829756d074e1fa70ae429bb52dae3
I20260810 22:34:17.174175 25583 ts_tablet_manager.cc:1939] T c9d829756d074e1fa70ae429bb52dae3 P 83e9b7216a8040b3a3418e588ecf2959: Deleting consensus metadata
I20260810 22:34:17.175166 24922 catalog_manager.cc:4841] TS 83e9b7216a8040b3a3418e588ecf2959 (127.16.233.66:37385): tablet c9d829756d074e1fa70ae429bb52dae3 (table test-table [id=8fd6504594d349dc89ebdacf159097ac]) successfully deleted
I20260810 22:34:17.176395 25554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:17.176096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.65:39951 (local address 127.0.0.1:60530)
0810 22:34:17.176255 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:17.176309 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.65:39951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260810 22:34:17.176631 24923 catalog_manager.cc:4584] TS 599ede616cdd412fa79af64abd77799a (127.16.233.65:39951): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c9d829756d074e1fa70ae429bb52dae3: Network error: Client connection negotiation failed: client connection to 127.16.233.65:39951: connect: Connection refused (error 111)
I20260810 22:34:17.176666 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 25143
I20260810 22:34:17.186997 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 25296
I20260810 22:34:17.196954 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 24567
I20260810 22:34:17.207046 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 24687
2026-08-10T22:34:17Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7905 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:34:17 dist-test-slave-77v9 krb5kdc[25588](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:34:17 dist-test-slave-77v9 krb5kdc[25588](info): set up 2 sockets
Aug 10 22:34:17 dist-test-slave-77v9 krb5kdc[25588](info): commencing operation
krb5kdc: starting...
W20260810 22:34:18.440236 17317 mini_kdc.cc:121] Time spent starting KDC: real 1.204s user 0.003s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Aug 10 22:34:18 dist-test-slave-77v9 krb5kdc[25588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:34:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:34:18Z Disabled control of system clock
WARNING: no policy specified for oryx/127.16.233.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.233.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.233.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/krb5kdc/oryx_127.16.233.126.keytab.
Entry for principal oryx/127.16.233.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/krb5kdc/oryx_127.16.233.126.keytab.
I20260810 22:34:18.594218 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:45483
--webserver_interface=127.16.233.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/krb5kdc/oryx_127.16.233.126.keytab
--principal=oryx/127.16.233.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:39311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.126:45483
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:34:18.724486 25602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:34:18.724876 25602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:34:18.724927 25602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:34:18.729811 25602 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:34:18.729902 25602 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 22:34:18.729920 25602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:34:18.729939 25602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:34:18.729955 25602 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:34:18.729972 25602 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:34:18.735838 25602 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:39311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.126:45483
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/krb5kdc/oryx_127.16.233.126.keytab
--principal=oryx/127.16.233.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:45483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:34:18.737237 25602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:34:18.738411 25602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:34:18.748651 25608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:18.748651 25611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:18.748651 25609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:18.748695 25602 server_base.cc:1034] running on GCE node
I20260810 22:34:18.749708 25602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:34:18.751011 25602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:34:18.752230 25602 hybrid_clock.cc:648] HybridClock initialized: now 1786401258752182 us; error 65 us; skew 500 ppm
Aug 10 22:34:18 dist-test-slave-77v9 krb5kdc[25588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401258, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:34:18.760032 25602 init.cc:377] Logged in from keytab as oryx/127.16.233.126@KRBTEST.COM (short username oryx)
I20260810 22:34:18.761473 25602 webserver.cc:460] Webserver started at http://127.16.233.126:38379/ using document root <none> and password file <none>
I20260810 22:34:18.762197 25602 fs_manager.cc:359] Metadata directory not provided
I20260810 22:34:18.762315 25602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:34:18.762593 25602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:34:18.765307 25602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/master-0/data/instance:
uuid: "98283e123558435d982c4c04a9e1d6a0"
format_stamp: "Formatted at 2026-08-10 22:34:18 on dist-test-slave-77v9"
server_key: "5404ef8d7d77d24dc407d864068a15ed"
server_key_iv: "9d079b32880c8605913cf29c31700df7"
server_key_version: "encryptionkey@0"
I20260810 22:34:18.766089 25602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/master-0/wal/instance:
uuid: "98283e123558435d982c4c04a9e1d6a0"
format_stamp: "Formatted at 2026-08-10 22:34:18 on dist-test-slave-77v9"
server_key: "5404ef8d7d77d24dc407d864068a15ed"
server_key_iv: "9d079b32880c8605913cf29c31700df7"
server_key_version: "encryptionkey@0"
I20260810 22:34:18.771319 25602 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260810 22:34:18.774616 25618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:18.776101 25602 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 22:34:18.776291 25602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/master-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/master-0/wal
uuid: "98283e123558435d982c4c04a9e1d6a0"
format_stamp: "Formatted at 2026-08-10 22:34:18 on dist-test-slave-77v9"
server_key: "5404ef8d7d77d24dc407d864068a15ed"
server_key_iv: "9d079b32880c8605913cf29c31700df7"
server_key_version: "encryptionkey@0"
I20260810 22:34:18.776453 25602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:34:18.789714 25602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:34:18.793814 25602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:34:18.794163 25602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:34:18.803617 25602 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.126:45483
I20260810 22:34:18.803781 25691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.126:45483 every 8 connection(s)
I20260810 22:34:18.804775 25602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
I20260810 22:34:18.808116 25692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:34:18.813925 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 25602
I20260810 22:34:18.814090 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/master-0/wal/instance
I20260810 22:34:18.814437 17317 external_mini_cluster.cc:1428] Setting key 7e2ec5a7575df867ee2df24e2ca03fc7
I20260810 22:34:18.815634 25692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98283e123558435d982c4c04a9e1d6a0: Bootstrap starting.
I20260810 22:34:18.818275 25692 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98283e123558435d982c4c04a9e1d6a0: Neither blocks nor log segments found. Creating new log.
I20260810 22:34:18.819139 25692 log.cc:824] T 00000000000000000000000000000000 P 98283e123558435d982c4c04a9e1d6a0: Log is configured to *not* fsync() on all Append() calls
I20260810 22:34:18.821367 25692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98283e123558435d982c4c04a9e1d6a0: No bootstrap required, opened a new log
Aug 10 22:34:18 dist-test-slave-77v9 krb5kdc[25588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM
I20260810 22:34:18.825532 25692 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98283e123558435d982c4c04a9e1d6a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98283e123558435d982c4c04a9e1d6a0" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 45483 } }
I20260810 22:34:18.825862 25692 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98283e123558435d982c4c04a9e1d6a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:34:18.825948 25692 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98283e123558435d982c4c04a9e1d6a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98283e123558435d982c4c04a9e1d6a0, State: Initialized, Role: FOLLOWER
I20260810 22:34:18.826570 25692 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98283e123558435d982c4c04a9e1d6a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98283e123558435d982c4c04a9e1d6a0" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 45483 } }
I20260810 22:34:18.826794 25692 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98283e123558435d982c4c04a9e1d6a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:34:18.826896 25692 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98283e123558435d982c4c04a9e1d6a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:34:18.827008 25692 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98283e123558435d982c4c04a9e1d6a0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:34:18.828089 25692 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98283e123558435d982c4c04a9e1d6a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98283e123558435d982c4c04a9e1d6a0" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 45483 } }
I20260810 22:34:18.828517 25692 leader_election.cc:302] T 00000000000000000000000000000000 P 98283e123558435d982c4c04a9e1d6a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98283e123558435d982c4c04a9e1d6a0; no voters:
I20260810 22:34:18.828965 25692 leader_election.cc:288] T 00000000000000000000000000000000 P 98283e123558435d982c4c04a9e1d6a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:34:18.829108 25697 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98283e123558435d982c4c04a9e1d6a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:34:18.829404 25697 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98283e123558435d982c4c04a9e1d6a0 [term 1 LEADER]: Becoming Leader. State: Replica: 98283e123558435d982c4c04a9e1d6a0, State: Running, Role: LEADER
I20260810 22:34:18.829869 25697 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98283e123558435d982c4c04a9e1d6a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98283e123558435d982c4c04a9e1d6a0" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 45483 } }
I20260810 22:34:18.830559 25692 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98283e123558435d982c4c04a9e1d6a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:34:18.831944 25698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98283e123558435d982c4c04a9e1d6a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "98283e123558435d982c4c04a9e1d6a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98283e123558435d982c4c04a9e1d6a0" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 45483 } } }
I20260810 22:34:18.832106 25698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98283e123558435d982c4c04a9e1d6a0 [sys.catalog]: This master's current role is: LEADER
I20260810 22:34:18.832223 25695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:18.816165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45436 (local address 127.16.233.126:45483)
0810 22:34:18.816777 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:18.816803 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:18.816919 (+ 116us) server_negotiation.cc:407] Connection header received
0810 22:34:18.817729 (+ 810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:18.817750 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:18.818144 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:18.818554 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:18.819614 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:18.820797 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:18.821936 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:18.822515 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:18.822716 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:18.825916 (+ 3200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:18.825923 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:18.825946 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:18.825993 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:18.828098 (+ 2105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:18.828663 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:18.828678 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:18.828685 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:18.828889 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:18.829335 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:18.829338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:18.829340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:18.829888 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:18.830102 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:18.830506 (+ 404us) server_negotiation.cc:299] Negotiation successful
0810 22:34:18.831054 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":251,"threads_started":1}
I20260810 22:34:18.832631 25709 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:34:18.832568 25699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98283e123558435d982c4c04a9e1d6a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98283e123558435d982c4c04a9e1d6a0. Latest consensus state: current_term: 1 leader_uuid: "98283e123558435d982c4c04a9e1d6a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98283e123558435d982c4c04a9e1d6a0" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 45483 } } }
I20260810 22:34:18.833374 25699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98283e123558435d982c4c04a9e1d6a0 [sys.catalog]: This master's current role is: LEADER
I20260810 22:34:18.836253 25709 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:34:18.850275 25709 catalog_manager.cc:1347] Generated new cluster ID: e2a72364c8d042c4bf8fc9c8ddc6b29b
I20260810 22:34:18.850389 25709 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:34:18.866261 25709 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:34:18.867584 25709 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:34:18.885031 25709 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98283e123558435d982c4c04a9e1d6a0: Generated new TSK 0
I20260810 22:34:18.885918 25709 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.16.233.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.233.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.233.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/krb5kdc/oryx_127.16.233.65.keytab.
Entry for principal oryx/127.16.233.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/krb5kdc/oryx_127.16.233.65.keytab.
I20260810 22:34:18.933125 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.65:0
--local_ip_for_outbound_sockets=127.16.233.65
--webserver_interface=127.16.233.65
--webserver_port=0
--tserver_master_addrs=127.16.233.126:45483
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/krb5kdc/oryx_127.16.233.65.keytab
--principal=oryx/127.16.233.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:39311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:34:19.085141 25720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:34:19.085498 25720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:34:19.085551 25720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:34:19.090322 25720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:34:19.090423 25720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:34:19.090471 25720 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 22:34:19.090565 25720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.65
I20260810 22:34:19.096794 25720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:39311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/krb5kdc/oryx_127.16.233.65.keytab
--principal=oryx/127.16.233.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.233.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.233.126:45483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.65
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:34:19.098093 25720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:34:19.099234 25720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:34:19.110862 25730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:19.110862 25727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:19.110934 25728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:19.111302 25720 server_base.cc:1034] running on GCE node
I20260810 22:34:19.111810 25720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:34:19.112596 25720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:34:19.113812 25720 hybrid_clock.cc:648] HybridClock initialized: now 1786401259113787 us; error 46 us; skew 500 ppm
Aug 10 22:34:19 dist-test-slave-77v9 krb5kdc[25588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401259, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:34:19.117378 25720 init.cc:377] Logged in from keytab as oryx/127.16.233.65@KRBTEST.COM (short username oryx)
I20260810 22:34:19.118712 25720 webserver.cc:460] Webserver started at http://127.16.233.65:45295/ using document root <none> and password file <none>
I20260810 22:34:19.119620 25720 fs_manager.cc:359] Metadata directory not provided
I20260810 22:34:19.119706 25720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:34:19.119992 25720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:34:19.122488 25720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/ts-0/data/instance:
uuid: "bd1ece4b27ae4e6c9c715a0045bac33c"
format_stamp: "Formatted at 2026-08-10 22:34:19 on dist-test-slave-77v9"
server_key: "584cd8afac00b0c79ea79f0da5ae76e1"
server_key_iv: "9f265c6ee1a94e5bd11b6f639821ada5"
server_key_version: "encryptionkey@0"
I20260810 22:34:19.123250 25720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/ts-0/wal/instance:
uuid: "bd1ece4b27ae4e6c9c715a0045bac33c"
format_stamp: "Formatted at 2026-08-10 22:34:19 on dist-test-slave-77v9"
server_key: "584cd8afac00b0c79ea79f0da5ae76e1"
server_key_iv: "9f265c6ee1a94e5bd11b6f639821ada5"
server_key_version: "encryptionkey@0"
I20260810 22:34:19.127789 25720 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260810 22:34:19.130934 25737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:19.132678 25720 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 22:34:19.132969 25720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/ts-0/wal
uuid: "bd1ece4b27ae4e6c9c715a0045bac33c"
format_stamp: "Formatted at 2026-08-10 22:34:19 on dist-test-slave-77v9"
server_key: "584cd8afac00b0c79ea79f0da5ae76e1"
server_key_iv: "9f265c6ee1a94e5bd11b6f639821ada5"
server_key_version: "encryptionkey@0"
I20260810 22:34:19.133121 25720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:34:19.157763 25720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:34:19.162055 25720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:34:19.162309 25720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:34:19.164139 25720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:34:19.164225 25720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:19.164300 25720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:34:19.164352 25720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 22:34:19 dist-test-slave-77v9 krb5kdc[25588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401259, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.65@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM
I20260810 22:34:19.176728 25720 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.65:33405
I20260810 22:34:19.176748 25867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.65:33405 every 8 connection(s)
I20260810 22:34:19.177799 25720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
I20260810 22:34:19.179716 25695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:19.165166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:39449 (local address 127.16.233.126:45483)
0810 22:34:19.165347 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:19.165351 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:19.166565 (+ 1214us) server_negotiation.cc:407] Connection header received
0810 22:34:19.167737 (+ 1172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:19.167741 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:19.167809 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:19.167934 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:19.169282 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:19.169934 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:19.171336 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:19.171659 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:19.171805 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:19.175362 (+ 3557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:19.175366 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:19.175369 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:19.175410 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:19.177320 (+ 1910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:19.177900 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:19.177904 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:19.177906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:19.178002 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:19.178435 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:19.178438 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:19.178440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:19.178703 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:19.178818 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:19.179343 (+ 525us) server_negotiation.cc:299] Negotiation successful
0810 22:34:19.179511 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 22:34:19.180511 25747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:19.165791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:45483 (local address 127.16.233.65:39449)
0810 22:34:19.166370 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:19.166421 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:19.167481 (+ 1060us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:19.168142 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:19.168149 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:19.168532 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:19.169117 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:19.169133 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:19.170092 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:19.170095 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:19.171197 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:19.171208 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:19.171876 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:19.171883 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:19.172184 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:19.173071 (+ 887us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:19.173112 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:19.175143 (+ 2031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:34:19.177477 (+ 2334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:19.177483 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:19.177502 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:19.177783 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:19.178108 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:19.178112 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:19.178114 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:19.178306 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:19.178816 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:34:19.178823 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:34:19.179081 (+ 258us) client_negotiation.cc:769] Sending connection context
0810 22:34:19.179290 (+ 209us) client_negotiation.cc:240] Negotiation successful
0810 22:34:19.179637 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":141,"threads_started":1}
I20260810 22:34:19.182071 25868 heartbeater.cc:347] Connected to a master server at 127.16.233.126:45483
I20260810 22:34:19.182363 25868 heartbeater.cc:464] Registering TS with master...
I20260810 22:34:19.182960 25868 heartbeater.cc:511] Master 127.16.233.126:45483 requested a full tablet report, sending...
I20260810 22:34:19.184037 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 25720
I20260810 22:34:19.184183 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/ts-0/wal/instance
I20260810 22:34:19.184520 17317 external_mini_cluster.cc:1428] Setting key 7266f285862a9aedb48db5278f845ccb
I20260810 22:34:19.184938 25637 ts_manager.cc:194] Registered new tserver with Master: bd1ece4b27ae4e6c9c715a0045bac33c (127.16.233.65:33405)
I20260810 22:34:19.186743 25637 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.233.65@KRBTEST.COM'} at 127.16.233.65:39449
I20260810 22:34:19.193320 25695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:19.184948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:46239 (local address 127.16.233.126:45483)
0810 22:34:19.185140 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:19.185146 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:19.185531 (+ 385us) server_negotiation.cc:407] Connection header received
0810 22:34:19.185654 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:19.185658 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:19.185712 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:19.185789 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:19.186324 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:19.186855 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:19.187905 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:19.188225 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:19.188306 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:19.189851 (+ 1545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:19.189854 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:19.189857 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:19.189884 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:19.191554 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:19.192006 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:19.192010 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:19.192012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:19.192101 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:19.192399 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:19.192402 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:19.192404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:19.192590 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:19.192700 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:19.193017 (+ 317us) server_negotiation.cc:299] Negotiation successful
0810 22:34:19.193135 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260810 22:34:19.193341 25874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:19.185008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:45483 (local address 127.16.233.65:46239)
0810 22:34:19.185423 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:19.185439 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:19.185552 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:19.185908 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:19.185910 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:19.186096 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:19.186203 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:19.186210 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:19.187049 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:19.187052 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:19.187726 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:19.187737 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:19.188474 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:19.188477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:19.188565 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:19.189200 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:19.189215 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:19.189671 (+ 456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:34:19.191669 (+ 1998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:19.191674 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:19.191676 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:19.191896 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:19.192199 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:19.192202 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:19.192204 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:19.192298 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:19.192691 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:34:19.192694 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:34:19.192796 (+ 102us) client_negotiation.cc:769] Sending connection context
0810 22:34:19.192962 (+ 166us) client_negotiation.cc:240] Negotiation successful
0810 22:34:19.193126 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":136,"threads_started":1}
WARNING: no policy specified for oryx/127.16.233.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.233.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.233.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/krb5kdc/oryx_127.16.233.66.keytab.
Entry for principal oryx/127.16.233.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/krb5kdc/oryx_127.16.233.66.keytab.
I20260810 22:34:19.221920 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.66:0
--local_ip_for_outbound_sockets=127.16.233.66
--webserver_interface=127.16.233.66
--webserver_port=0
--tserver_master_addrs=127.16.233.126:45483
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/krb5kdc/oryx_127.16.233.66.keytab
--principal=oryx/127.16.233.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:39311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:34:19.349675 25877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:34:19.349999 25877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:34:19.350050 25877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:34:19.354434 25877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:34:19.354521 25877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:34:19.354565 25877 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 22:34:19.354600 25877 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.66
I20260810 22:34:19.360423 25877 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:39311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/krb5kdc/oryx_127.16.233.66.keytab
--principal=oryx/127.16.233.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.233.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.233.126:45483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.66
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:34:19.361666 25877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:34:19.362907 25877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:34:19.374116 25885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:19.374063 25883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:19.374063 25882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:19.374641 25877 server_base.cc:1034] running on GCE node
I20260810 22:34:19.375131 25877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:34:19.375869 25877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:34:19.377072 25877 hybrid_clock.cc:648] HybridClock initialized: now 1786401259377048 us; error 39 us; skew 500 ppm
Aug 10 22:34:19 dist-test-slave-77v9 krb5kdc[25588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401259, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:34:19.380684 25877 init.cc:377] Logged in from keytab as oryx/127.16.233.66@KRBTEST.COM (short username oryx)
I20260810 22:34:19.381990 25877 webserver.cc:460] Webserver started at http://127.16.233.66:42473/ using document root <none> and password file <none>
I20260810 22:34:19.382618 25877 fs_manager.cc:359] Metadata directory not provided
I20260810 22:34:19.382764 25877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:34:19.383041 25877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:34:19.385974 25877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/ts-1/data/instance:
uuid: "c33ed31d8faf4bc89a305efea7fa55a9"
format_stamp: "Formatted at 2026-08-10 22:34:19 on dist-test-slave-77v9"
server_key: "8e3c2ec73daee62218c5a5bb2f462114"
server_key_iv: "bae73eb254f0ffde2b15bb6e0148bb3c"
server_key_version: "encryptionkey@0"
I20260810 22:34:19.386709 25877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/ts-1/wal/instance:
uuid: "c33ed31d8faf4bc89a305efea7fa55a9"
format_stamp: "Formatted at 2026-08-10 22:34:19 on dist-test-slave-77v9"
server_key: "8e3c2ec73daee62218c5a5bb2f462114"
server_key_iv: "bae73eb254f0ffde2b15bb6e0148bb3c"
server_key_version: "encryptionkey@0"
I20260810 22:34:19.391245 25877 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 22:34:19.394394 25892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:19.395857 25877 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 22:34:19.396049 25877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/ts-1/wal
uuid: "c33ed31d8faf4bc89a305efea7fa55a9"
format_stamp: "Formatted at 2026-08-10 22:34:19 on dist-test-slave-77v9"
server_key: "8e3c2ec73daee62218c5a5bb2f462114"
server_key_iv: "bae73eb254f0ffde2b15bb6e0148bb3c"
server_key_version: "encryptionkey@0"
I20260810 22:34:19.396188 25877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:34:19.424412 25877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:34:19.428882 25877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:34:19.429176 25877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:34:19.431497 25877 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:34:19.431576 25877 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:19.431641 25877 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:34:19.431668 25877 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:19.444439 25877 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.66:39183
I20260810 22:34:19.444581 26019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.66:39183 every 8 connection(s)
Aug 10 22:34:19 dist-test-slave-77v9 krb5kdc[25588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401259, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.66@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM
I20260810 22:34:19.445699 25877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
I20260810 22:34:19.451086 25695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:19.433543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:49895 (local address 127.16.233.126:45483)
0810 22:34:19.433808 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:19.433814 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:19.435743 (+ 1929us) server_negotiation.cc:407] Connection header received
0810 22:34:19.437095 (+ 1352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:19.437099 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:19.437186 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:19.437279 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:19.439057 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:19.439970 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:19.442222 (+ 2252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:19.442657 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:19.442808 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:19.446271 (+ 3463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:19.446276 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:19.446279 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:19.446316 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:19.448103 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:19.448993 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:19.448997 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:19.449000 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:19.449127 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:19.449585 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:19.449589 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:19.449591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:19.449901 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:19.450045 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:19.450673 (+ 628us) server_negotiation.cc:299] Negotiation successful
0810 22:34:19.450845 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260810 22:34:19.451417 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 25877
I20260810 22:34:19.451589 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/ts-1/wal/instance
I20260810 22:34:19.451894 17317 external_mini_cluster.cc:1428] Setting key a41604ed1784cc0832ef8f91056c0b3e
I20260810 22:34:19.452243 25910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:19.434998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:45483 (local address 127.16.233.66:49895)
0810 22:34:19.435563 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:19.435621 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:19.436815 (+ 1194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:19.437495 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:19.437507 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:19.438082 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:19.438859 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:19.438902 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:19.440163 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:19.440169 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:19.441999 (+ 1830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:19.442020 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:19.442899 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:19.442909 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:19.443183 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:19.443991 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:19.444021 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:19.446027 (+ 2006us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:34:19.448268 (+ 2241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:19.448275 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:19.448292 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:19.448621 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:19.449269 (+ 648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:19.449273 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:19.449276 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:19.449457 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:19.450033 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:34:19.450041 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:34:19.450367 (+ 326us) client_negotiation.cc:769] Sending connection context
0810 22:34:19.450589 (+ 222us) client_negotiation.cc:240] Negotiation successful
0810 22:34:19.450990 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":164,"threads_started":1}
I20260810 22:34:19.453934 26020 heartbeater.cc:347] Connected to a master server at 127.16.233.126:45483
I20260810 22:34:19.454136 26020 heartbeater.cc:464] Registering TS with master...
I20260810 22:34:19.454828 26020 heartbeater.cc:511] Master 127.16.233.126:45483 requested a full tablet report, sending...
I20260810 22:34:19.456219 25637 ts_manager.cc:194] Registered new tserver with Master: c33ed31d8faf4bc89a305efea7fa55a9 (127.16.233.66:39183)
I20260810 22:34:19.457170 25637 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.233.66@KRBTEST.COM'} at 127.16.233.66:49895
WARNING: no policy specified for oryx/127.16.233.67@KRBTEST.COM; defaulting to no policy
I20260810 22:34:19.467155 26027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:19.456528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:45483 (local address 127.16.233.66:45835)
0810 22:34:19.457020 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:19.457036 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:19.457155 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:19.457505 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:19.457511 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:19.457800 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:19.457951 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:19.457961 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:19.459322 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:19.459329 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:19.460519 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:19.460536 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:19.461225 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:19.461234 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:19.461395 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:19.462042 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:19.462067 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:19.462640 (+ 573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:34:19.464976 (+ 2336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:19.464980 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:19.464983 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:19.465298 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:19.465728 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:19.465732 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:19.465734 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:19.465841 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:19.466565 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:34:19.466567 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:34:19.466664 (+ 97us) client_negotiation.cc:769] Sending connection context
0810 22:34:19.466835 (+ 171us) client_negotiation.cc:240] Negotiation successful
0810 22:34:19.466983 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":112,"threads_started":1}
I20260810 22:34:19.467237 25695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:19.456556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:45835 (local address 127.16.233.126:45483)
0810 22:34:19.456721 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:19.456729 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:19.457128 (+ 399us) server_negotiation.cc:407] Connection header received
0810 22:34:19.457269 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:19.457272 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:19.457343 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:19.457463 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:19.458207 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:19.459126 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:19.460677 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:19.461070 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:19.461172 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:19.462892 (+ 1720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:19.462896 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:19.462898 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:19.462926 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:19.464678 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:19.465458 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:19.465463 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:19.465465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:19.465595 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:19.466065 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:19.466068 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:19.466069 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:19.466397 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:19.466569 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:19.466917 (+ 348us) server_negotiation.cc:299] Negotiation successful
0810 22:34:19.467070 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.233.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.233.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/krb5kdc/oryx_127.16.233.67.keytab.
Entry for principal oryx/127.16.233.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/krb5kdc/oryx_127.16.233.67.keytab.
I20260810 22:34:19.489895 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.67:0
--local_ip_for_outbound_sockets=127.16.233.67
--webserver_interface=127.16.233.67
--webserver_port=0
--tserver_master_addrs=127.16.233.126:45483
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/krb5kdc/oryx_127.16.233.67.keytab
--principal=oryx/127.16.233.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:39311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:34:19.612990 26029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:34:19.613336 26029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:34:19.613420 26029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:34:19.617763 26029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:34:19.617980 26029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:34:19.618221 26029 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 22:34:19.618321 26029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.67
I20260810 22:34:19.624760 26029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:39311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/krb5kdc/oryx_127.16.233.67.keytab
--principal=oryx/127.16.233.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.233.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.233.126:45483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.67
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:34:19.626147 26029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:34:19.627324 26029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:34:19.638236 26034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:19.638207 26035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:19.638595 26037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:19.638890 26029 server_base.cc:1034] running on GCE node
I20260810 22:34:19.639407 26029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:34:19.640137 26029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:34:19.641460 26029 hybrid_clock.cc:648] HybridClock initialized: now 1786401259641432 us; error 57 us; skew 500 ppm
Aug 10 22:34:19 dist-test-slave-77v9 krb5kdc[25588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401259, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:34:19.644644 26029 init.cc:377] Logged in from keytab as oryx/127.16.233.67@KRBTEST.COM (short username oryx)
I20260810 22:34:19.645821 26029 webserver.cc:460] Webserver started at http://127.16.233.67:43817/ using document root <none> and password file <none>
I20260810 22:34:19.646425 26029 fs_manager.cc:359] Metadata directory not provided
I20260810 22:34:19.646513 26029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:34:19.646703 26029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:34:19.649163 26029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/ts-2/data/instance:
uuid: "4bcacb667ac14937b041f7d4694b907b"
format_stamp: "Formatted at 2026-08-10 22:34:19 on dist-test-slave-77v9"
server_key: "cd526de2a59675d027c2f08e83e679d5"
server_key_iv: "880cb69979d55a3dc51a77ad5d1015cb"
server_key_version: "encryptionkey@0"
I20260810 22:34:19.649785 26029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/ts-2/wal/instance:
uuid: "4bcacb667ac14937b041f7d4694b907b"
format_stamp: "Formatted at 2026-08-10 22:34:19 on dist-test-slave-77v9"
server_key: "cd526de2a59675d027c2f08e83e679d5"
server_key_iv: "880cb69979d55a3dc51a77ad5d1015cb"
server_key_version: "encryptionkey@0"
I20260810 22:34:19.654361 26029 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260810 22:34:19.657277 26044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:19.658789 26029 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260810 22:34:19.658916 26029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/ts-2/wal
uuid: "4bcacb667ac14937b041f7d4694b907b"
format_stamp: "Formatted at 2026-08-10 22:34:19 on dist-test-slave-77v9"
server_key: "cd526de2a59675d027c2f08e83e679d5"
server_key_iv: "880cb69979d55a3dc51a77ad5d1015cb"
server_key_version: "encryptionkey@0"
I20260810 22:34:19.659080 26029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:34:19.683985 26029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:34:19.687784 26029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:34:19.688032 26029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:34:19.689848 26029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:34:19.689921 26029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:19.690351 26029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:34:19.690449 26029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 22:34:19 dist-test-slave-77v9 krb5kdc[25588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401259, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.67@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM
I20260810 22:34:19.702904 26029 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.67:41239
I20260810 22:34:19.703482 26176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.67:41239 every 8 connection(s)
I20260810 22:34:19.704005 26029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
I20260810 22:34:19.707298 25695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:19.691139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:51621 (local address 127.16.233.126:45483)
0810 22:34:19.691312 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:19.691317 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:19.692658 (+ 1341us) server_negotiation.cc:407] Connection header received
0810 22:34:19.694032 (+ 1374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:19.694036 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:19.694108 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:19.694158 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:19.695814 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:19.696526 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:19.698788 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:19.699240 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:19.699423 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:19.703140 (+ 3717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:19.703144 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:19.703149 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:19.703192 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:19.705062 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:19.705626 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:19.705635 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:19.705637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:19.705721 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:19.706090 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:19.706094 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:19.706096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:19.706310 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:19.706418 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:19.706973 (+ 555us) server_negotiation.cc:299] Negotiation successful
0810 22:34:19.707116 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 22:34:19.708196 26054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:19.691794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:45483 (local address 127.16.233.67:51621)
0810 22:34:19.692439 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:19.692511 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:19.693717 (+ 1206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:19.694587 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:19.694594 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:19.695002 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:19.695539 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:19.695554 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:19.696687 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:19.696691 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:19.698563 (+ 1872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:19.698583 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:19.699459 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:19.699465 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:19.699714 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:19.700526 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:19.700545 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:19.702897 (+ 2352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:34:19.705190 (+ 2293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:19.705197 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:19.705212 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:19.705499 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:19.705817 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:19.705821 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:19.705823 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:19.705978 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:19.706419 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:34:19.706427 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:34:19.706702 (+ 275us) client_negotiation.cc:769] Sending connection context
0810 22:34:19.706913 (+ 211us) client_negotiation.cc:240] Negotiation successful
0810 22:34:19.707267 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":178,"threads_started":1}
I20260810 22:34:19.709228 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 26029
I20260810 22:34:19.709406 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/ts-2/wal/instance
I20260810 22:34:19.709754 17317 external_mini_cluster.cc:1428] Setting key e77847c88fbc5ffa0de8daa4a9cc53ff
I20260810 22:34:19.710124 26177 heartbeater.cc:347] Connected to a master server at 127.16.233.126:45483
I20260810 22:34:19.710295 26177 heartbeater.cc:464] Registering TS with master...
I20260810 22:34:19.711078 26177 heartbeater.cc:511] Master 127.16.233.126:45483 requested a full tablet report, sending...
I20260810 22:34:19.712376 25637 ts_manager.cc:194] Registered new tserver with Master: 4bcacb667ac14937b041f7d4694b907b (127.16.233.67:41239)
I20260810 22:34:19.713195 25637 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.233.67@KRBTEST.COM'} at 127.16.233.67:51621
I20260810 22:34:19.722054 26183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:19.712752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:45483 (local address 127.16.233.67:41577)
0810 22:34:19.713133 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:19.713147 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:19.713289 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:19.713589 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:19.713592 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:19.713891 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:19.714043 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:19.714051 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:19.715294 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:19.715298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:19.715990 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:19.716000 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:19.716539 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:19.716542 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:19.716623 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:19.717446 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:19.717469 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:19.717873 (+ 404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:34:19.719999 (+ 2126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:19.720002 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:19.720003 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:19.720240 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:19.720607 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:19.720609 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:19.720611 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:19.720730 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:19.721479 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:34:19.721482 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:34:19.721584 (+ 102us) client_negotiation.cc:769] Sending connection context
0810 22:34:19.721701 (+ 117us) client_negotiation.cc:240] Negotiation successful
0810 22:34:19.721860 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":175,"threads_started":1}
I20260810 22:34:19.722422 25695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:19.712737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:41577 (local address 127.16.233.126:45483)
0810 22:34:19.712920 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:19.712933 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:19.713233 (+ 300us) server_negotiation.cc:407] Connection header received
0810 22:34:19.713404 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:19.713407 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:19.713456 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:19.713549 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:19.714197 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:19.715125 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:19.716140 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:19.716394 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:19.716502 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:19.718065 (+ 1563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:19.718068 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:19.718070 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:19.718106 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:19.719823 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:19.720369 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:19.720372 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:19.720374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:19.720476 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:19.720984 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:19.720987 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:19.720988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:19.721338 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:19.721451 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:19.721932 (+ 481us) server_negotiation.cc:299] Negotiation successful
0810 22:34:19.722185 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 22:34:19.724532 17317 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 22:34:19 dist-test-slave-77v9 krb5kdc[25588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401259, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 22:34:19 dist-test-slave-77v9 krb5kdc[25588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:34:19 dist-test-slave-77v9 krb5kdc[25588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
W20260810 22:34:19.755793 25695 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:34:19.748056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45442 (local address 127.16.233.126:45483)
0810 22:34:19.748208 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:19.748212 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:19.748228 (+ 16us) server_negotiation.cc:407] Connection header received
0810 22:34:19.748276 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:19.748279 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:19.748322 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:19.748415 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:19.749486 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:19.750120 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:19.751128 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:19.751381 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:19.751491 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:19.755646 (+ 4155us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45442: BlockingRecv error: recv got EOF from 127.0.0.1:45442 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 10 22:34:19 dist-test-slave-77v9 krb5kdc[25588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401259, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM
I20260810 22:34:19.767072 25695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:19.757569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45452 (local address 127.16.233.126:45483)
0810 22:34:19.757844 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:19.757851 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:19.757888 (+ 37us) server_negotiation.cc:407] Connection header received
0810 22:34:19.758058 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:19.758061 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:19.758106 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:19.758198 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:19.758856 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:19.759396 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:19.760230 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:19.760465 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:19.760583 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:19.763625 (+ 3042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:19.763629 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:19.763631 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:19.763664 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:19.765254 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:19.765715 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:19.765718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:19.765719 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:19.765825 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:19.766177 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:19.766180 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:19.766181 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:19.766396 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:19.766517 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:19.766794 (+ 277us) server_negotiation.cc:299] Negotiation successful
0810 22:34:19.766920 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 22:34:19.770936 25637 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45452:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 22:34:19.774696 25637 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:34:19.793572 26200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:19.786020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.67:41239 (local address 127.0.0.1:56890)
0810 22:34:19.787321 (+ 1301us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:19.787395 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:19.787630 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:19.788492 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:19.788496 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:19.788518 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:34:19.788863 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:19.788875 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:19.790242 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:19.790245 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:19.791670 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:19.791683 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:19.792840 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:19.792844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:19.792980 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:19.793033 (+ 53us) client_negotiation.cc:769] Sending connection context
0810 22:34:19.793174 (+ 141us) client_negotiation.cc:240] Negotiation successful
0810 22:34:19.793392 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1099,"mutex_wait_us":191,"thread_start_us":107,"threads_started":1}
I20260810 22:34:19.793640 26203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:19.786534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56890 (local address 127.16.233.67:41239)
0810 22:34:19.787350 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:19.787365 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:19.787495 (+ 130us) server_negotiation.cc:407] Connection header received
0810 22:34:19.787842 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:19.787850 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:19.788100 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:19.788250 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:34:19.789015 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:19.790120 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:19.791822 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:19.792639 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:19.792767 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:19.792995 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:19.793228 (+ 233us) server_negotiation.cc:299] Negotiation successful
0810 22:34:19.793352 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":705,"thread_start_us":118,"threads_started":1}
I20260810 22:34:19.794436 26202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:19.786429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34228 (local address 127.16.233.66:39183)
0810 22:34:19.786957 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:19.787336 (+ 379us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:19.787632 (+ 296us) server_negotiation.cc:407] Connection header received
0810 22:34:19.787860 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:19.787866 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:19.788100 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:19.788250 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:34:19.789114 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:19.789992 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:19.792261 (+ 2269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:19.793214 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:19.793329 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:19.793495 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:19.794066 (+ 571us) server_negotiation.cc:299] Negotiation successful
0810 22:34:19.794264 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":106,"threads_started":1}
I20260810 22:34:19.794545 26204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:19.786910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.65:33405 (local address 127.0.0.1:57154)
0810 22:34:19.787681 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:19.787695 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:19.787868 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:19.788417 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:19.788423 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:19.788449 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:34:19.788613 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:19.788621 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:19.790229 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:19.790233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:19.792164 (+ 1931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:19.792176 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:19.793579 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:19.793585 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:19.793688 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:19.794228 (+ 540us) client_negotiation.cc:769] Sending connection context
0810 22:34:19.794307 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 22:34:19.794402 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"mutex_wait_us":275,"thread_start_us":124,"threads_started":1}
I20260810 22:34:19.794957 26201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:19.786303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.66:39183 (local address 127.0.0.1:34228)
0810 22:34:19.787531 (+ 1228us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:19.787545 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:19.787704 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:19.788501 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:19.788504 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:19.788556 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:34:19.788897 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:19.788908 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:19.790161 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:19.790167 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:19.792087 (+ 1920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:19.792099 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:19.793662 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:19.793667 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:19.793819 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:19.793852 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 22:34:19.794664 (+ 812us) client_negotiation.cc:240] Negotiation successful
0810 22:34:19.794775 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1153,"mutex_wait_us":301,"spinlock_wait_cycles":768,"thread_start_us":202,"threads_started":1}
I20260810 22:34:19.795140 26205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:19.787081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57154 (local address 127.16.233.65:33405)
0810 22:34:19.787519 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:19.787525 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:19.787788 (+ 263us) server_negotiation.cc:407] Connection header received
0810 22:34:19.788078 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:19.788083 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:19.788251 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:19.788427 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:34:19.788739 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:19.790068 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:19.792340 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:19.793258 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:19.793368 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:19.793511 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:19.794871 (+ 1360us) server_negotiation.cc:299] Negotiation successful
0810 22:34:19.794996 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":69,"threads_started":1}
I20260810 22:34:19.797601 26101 tablet_service.cc:1467] Processing CreateTablet for tablet e01af2908eae46938a66d9db4dcb54e4 (DEFAULT_TABLE table=test-table [id=01597677cdc94000ada79dd30c5986ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:34:19.797937 25941 tablet_service.cc:1467] Processing CreateTablet for tablet e01af2908eae46938a66d9db4dcb54e4 (DEFAULT_TABLE table=test-table [id=01597677cdc94000ada79dd30c5986ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:34:19.797936 25793 tablet_service.cc:1467] Processing CreateTablet for tablet e01af2908eae46938a66d9db4dcb54e4 (DEFAULT_TABLE table=test-table [id=01597677cdc94000ada79dd30c5986ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:34:19.799021 26101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e01af2908eae46938a66d9db4dcb54e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:34:19.799063 25793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e01af2908eae46938a66d9db4dcb54e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:34:19.799063 25941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e01af2908eae46938a66d9db4dcb54e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:34:19.808521 26206 tablet_bootstrap.cc:492] T e01af2908eae46938a66d9db4dcb54e4 P bd1ece4b27ae4e6c9c715a0045bac33c: Bootstrap starting.
I20260810 22:34:19.809128 26208 tablet_bootstrap.cc:492] T e01af2908eae46938a66d9db4dcb54e4 P 4bcacb667ac14937b041f7d4694b907b: Bootstrap starting.
I20260810 22:34:19.810421 26207 tablet_bootstrap.cc:492] T e01af2908eae46938a66d9db4dcb54e4 P c33ed31d8faf4bc89a305efea7fa55a9: Bootstrap starting.
I20260810 22:34:19.811560 26208 tablet_bootstrap.cc:654] T e01af2908eae46938a66d9db4dcb54e4 P 4bcacb667ac14937b041f7d4694b907b: Neither blocks nor log segments found. Creating new log.
I20260810 22:34:19.811570 26206 tablet_bootstrap.cc:654] T e01af2908eae46938a66d9db4dcb54e4 P bd1ece4b27ae4e6c9c715a0045bac33c: Neither blocks nor log segments found. Creating new log.
I20260810 22:34:19.812505 26208 log.cc:824] T e01af2908eae46938a66d9db4dcb54e4 P 4bcacb667ac14937b041f7d4694b907b: Log is configured to *not* fsync() on all Append() calls
I20260810 22:34:19.812541 26206 log.cc:824] T e01af2908eae46938a66d9db4dcb54e4 P bd1ece4b27ae4e6c9c715a0045bac33c: Log is configured to *not* fsync() on all Append() calls
I20260810 22:34:19.813155 26207 tablet_bootstrap.cc:654] T e01af2908eae46938a66d9db4dcb54e4 P c33ed31d8faf4bc89a305efea7fa55a9: Neither blocks nor log segments found. Creating new log.
I20260810 22:34:19.814939 26208 tablet_bootstrap.cc:492] T e01af2908eae46938a66d9db4dcb54e4 P 4bcacb667ac14937b041f7d4694b907b: No bootstrap required, opened a new log
I20260810 22:34:19.814957 26206 tablet_bootstrap.cc:492] T e01af2908eae46938a66d9db4dcb54e4 P bd1ece4b27ae4e6c9c715a0045bac33c: No bootstrap required, opened a new log
I20260810 22:34:19.815166 26208 ts_tablet_manager.cc:1397] T e01af2908eae46938a66d9db4dcb54e4 P 4bcacb667ac14937b041f7d4694b907b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 22:34:19.815167 26206 ts_tablet_manager.cc:1397] T e01af2908eae46938a66d9db4dcb54e4 P bd1ece4b27ae4e6c9c715a0045bac33c: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 22:34:19.815533 26207 log.cc:824] T e01af2908eae46938a66d9db4dcb54e4 P c33ed31d8faf4bc89a305efea7fa55a9: Log is configured to *not* fsync() on all Append() calls
I20260810 22:34:19.819250 26207 tablet_bootstrap.cc:492] T e01af2908eae46938a66d9db4dcb54e4 P c33ed31d8faf4bc89a305efea7fa55a9: No bootstrap required, opened a new log
I20260810 22:34:19.819522 26207 ts_tablet_manager.cc:1397] T e01af2908eae46938a66d9db4dcb54e4 P c33ed31d8faf4bc89a305efea7fa55a9: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260810 22:34:19.820461 26206 raft_consensus.cc:348] T e01af2908eae46938a66d9db4dcb54e4 P bd1ece4b27ae4e6c9c715a0045bac33c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bcacb667ac14937b041f7d4694b907b" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 41239 } } peers { permanent_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39183 } } peers { permanent_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33405 } }
I20260810 22:34:19.820855 26206 raft_consensus.cc:374] T e01af2908eae46938a66d9db4dcb54e4 P bd1ece4b27ae4e6c9c715a0045bac33c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:34:19.820920 26206 raft_consensus.cc:729] T e01af2908eae46938a66d9db4dcb54e4 P bd1ece4b27ae4e6c9c715a0045bac33c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd1ece4b27ae4e6c9c715a0045bac33c, State: Initialized, Role: FOLLOWER
I20260810 22:34:19.821542 26206 consensus_queue.cc:260] T e01af2908eae46938a66d9db4dcb54e4 P bd1ece4b27ae4e6c9c715a0045bac33c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bcacb667ac14937b041f7d4694b907b" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 41239 } } peers { permanent_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39183 } } peers { permanent_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33405 } }
I20260810 22:34:19.822762 26206 ts_tablet_manager.cc:1428] T e01af2908eae46938a66d9db4dcb54e4 P bd1ece4b27ae4e6c9c715a0045bac33c: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260810 22:34:19.823179 25868 heartbeater.cc:503] Master 127.16.233.126:45483 was elected leader, sending a full tablet report...
I20260810 22:34:19.823487 26208 raft_consensus.cc:348] T e01af2908eae46938a66d9db4dcb54e4 P 4bcacb667ac14937b041f7d4694b907b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bcacb667ac14937b041f7d4694b907b" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 41239 } } peers { permanent_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39183 } } peers { permanent_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33405 } }
I20260810 22:34:19.823818 26208 raft_consensus.cc:374] T e01af2908eae46938a66d9db4dcb54e4 P 4bcacb667ac14937b041f7d4694b907b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:34:19.823879 26208 raft_consensus.cc:729] T e01af2908eae46938a66d9db4dcb54e4 P 4bcacb667ac14937b041f7d4694b907b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bcacb667ac14937b041f7d4694b907b, State: Initialized, Role: FOLLOWER
I20260810 22:34:19.824385 26208 consensus_queue.cc:260] T e01af2908eae46938a66d9db4dcb54e4 P 4bcacb667ac14937b041f7d4694b907b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bcacb667ac14937b041f7d4694b907b" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 41239 } } peers { permanent_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39183 } } peers { permanent_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33405 } }
I20260810 22:34:19.824293 26207 raft_consensus.cc:348] T e01af2908eae46938a66d9db4dcb54e4 P c33ed31d8faf4bc89a305efea7fa55a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bcacb667ac14937b041f7d4694b907b" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 41239 } } peers { permanent_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39183 } } peers { permanent_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33405 } }
I20260810 22:34:19.824641 26207 raft_consensus.cc:374] T e01af2908eae46938a66d9db4dcb54e4 P c33ed31d8faf4bc89a305efea7fa55a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:34:19.824718 26207 raft_consensus.cc:729] T e01af2908eae46938a66d9db4dcb54e4 P c33ed31d8faf4bc89a305efea7fa55a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c33ed31d8faf4bc89a305efea7fa55a9, State: Initialized, Role: FOLLOWER
I20260810 22:34:19.825358 26214 raft_consensus.cc:482] T e01af2908eae46938a66d9db4dcb54e4 P bd1ece4b27ae4e6c9c715a0045bac33c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:34:19.825582 26214 raft_consensus.cc:504] T e01af2908eae46938a66d9db4dcb54e4 P bd1ece4b27ae4e6c9c715a0045bac33c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bcacb667ac14937b041f7d4694b907b" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 41239 } } peers { permanent_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39183 } } peers { permanent_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33405 } }
I20260810 22:34:19.825721 26208 ts_tablet_manager.cc:1428] T e01af2908eae46938a66d9db4dcb54e4 P 4bcacb667ac14937b041f7d4694b907b: Time spent starting tablet: real 0.010s user 0.008s sys 0.000s
I20260810 22:34:19.826198 26177 heartbeater.cc:503] Master 127.16.233.126:45483 was elected leader, sending a full tablet report...
I20260810 22:34:19.827203 26214 leader_election.cc:288] T e01af2908eae46938a66d9db4dcb54e4 P bd1ece4b27ae4e6c9c715a0045bac33c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bcacb667ac14937b041f7d4694b907b (127.16.233.67:41239), c33ed31d8faf4bc89a305efea7fa55a9 (127.16.233.66:39183)
I20260810 22:34:19.829465 26207 consensus_queue.cc:260] T e01af2908eae46938a66d9db4dcb54e4 P c33ed31d8faf4bc89a305efea7fa55a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bcacb667ac14937b041f7d4694b907b" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 41239 } } peers { permanent_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39183 } } peers { permanent_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33405 } }
I20260810 22:34:19.830915 26207 ts_tablet_manager.cc:1428] T e01af2908eae46938a66d9db4dcb54e4 P c33ed31d8faf4bc89a305efea7fa55a9: Time spent starting tablet: real 0.011s user 0.008s sys 0.000s
I20260810 22:34:19.831442 26020 heartbeater.cc:503] Master 127.16.233.126:45483 was elected leader, sending a full tablet report...
I20260810 22:34:19.833652 26215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:19.827475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.67:41239 (local address 127.16.233.65:52855)
0810 22:34:19.828195 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:19.828215 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:19.828348 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:19.828788 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:19.828791 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:19.828842 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:34:19.828996 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:19.829007 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:19.830283 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:19.830287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:19.831788 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:19.831800 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:19.833129 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:19.833133 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:19.833256 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:19.833282 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 22:34:19.833378 (+ 96us) client_negotiation.cc:240] Negotiation successful
0810 22:34:19.833463 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":168,"threads_started":1}
I20260810 22:34:19.833668 26203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:19.827991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:52855 (local address 127.16.233.67:41239)
0810 22:34:19.828134 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:19.828139 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:19.828306 (+ 167us) server_negotiation.cc:407] Connection header received
0810 22:34:19.828476 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:19.828482 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:19.828554 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:19.828643 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:34:19.829156 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:19.829909 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:19.832138 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:19.832778 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:19.832950 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:19.833010 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:19.833406 (+ 396us) server_negotiation.cc:299] Negotiation successful
0810 22:34:19.833500 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 22:34:19.834465 26123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e01af2908eae46938a66d9db4dcb54e4" candidate_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bcacb667ac14937b041f7d4694b907b" is_pre_election: true
I20260810 22:34:19.834823 26123 raft_consensus.cc:2454] T e01af2908eae46938a66d9db4dcb54e4 P 4bcacb667ac14937b041f7d4694b907b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd1ece4b27ae4e6c9c715a0045bac33c in term 0.
I20260810 22:34:19.835402 25740 leader_election.cc:302] T e01af2908eae46938a66d9db4dcb54e4 P bd1ece4b27ae4e6c9c715a0045bac33c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bcacb667ac14937b041f7d4694b907b, bd1ece4b27ae4e6c9c715a0045bac33c; no voters:
I20260810 22:34:19.835777 26214 raft_consensus.cc:2781] T e01af2908eae46938a66d9db4dcb54e4 P bd1ece4b27ae4e6c9c715a0045bac33c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:34:19.835942 26214 raft_consensus.cc:482] T e01af2908eae46938a66d9db4dcb54e4 P bd1ece4b27ae4e6c9c715a0045bac33c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:34:19.836012 26214 raft_consensus.cc:3037] T e01af2908eae46938a66d9db4dcb54e4 P bd1ece4b27ae4e6c9c715a0045bac33c [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:34:19.836714 26216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:19.827743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.66:39183 (local address 127.16.233.65:59681)
0810 22:34:19.831733 (+ 3990us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:19.831753 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:19.831900 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:19.832439 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:19.832442 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:19.832468 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:34:19.832596 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:19.832604 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:19.833808 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:19.833815 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:19.835113 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:19.835130 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:19.836161 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:19.836165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:19.836304 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:19.836338 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 22:34:19.836449 (+ 111us) client_negotiation.cc:240] Negotiation successful
0810 22:34:19.836541 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3900,"thread_start_us":130,"threads_started":1}
I20260810 22:34:19.836721 26202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:19.827850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:59681 (local address 127.16.233.66:39183)
0810 22:34:19.828023 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:19.828030 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:19.831871 (+ 3841us) server_negotiation.cc:407] Connection header received
0810 22:34:19.832151 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:19.832155 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:19.832212 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:19.832296 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:34:19.832844 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:19.833651 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:19.835281 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:19.835855 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:19.835971 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:19.836038 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:19.836529 (+ 491us) server_negotiation.cc:299] Negotiation successful
0810 22:34:19.836623 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 22:34:19.837489 25963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e01af2908eae46938a66d9db4dcb54e4" candidate_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" is_pre_election: true
I20260810 22:34:19.837641 26214 raft_consensus.cc:504] T e01af2908eae46938a66d9db4dcb54e4 P bd1ece4b27ae4e6c9c715a0045bac33c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bcacb667ac14937b041f7d4694b907b" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 41239 } } peers { permanent_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39183 } } peers { permanent_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33405 } }
I20260810 22:34:19.837883 25963 raft_consensus.cc:2454] T e01af2908eae46938a66d9db4dcb54e4 P c33ed31d8faf4bc89a305efea7fa55a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd1ece4b27ae4e6c9c715a0045bac33c in term 0.
I20260810 22:34:19.838183 26214 leader_election.cc:288] T e01af2908eae46938a66d9db4dcb54e4 P bd1ece4b27ae4e6c9c715a0045bac33c [CANDIDATE]: Term 1 election: Requested vote from peers 4bcacb667ac14937b041f7d4694b907b (127.16.233.67:41239), c33ed31d8faf4bc89a305efea7fa55a9 (127.16.233.66:39183)
I20260810 22:34:19.838663 26123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e01af2908eae46938a66d9db4dcb54e4" candidate_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bcacb667ac14937b041f7d4694b907b"
I20260810 22:34:19.838841 26123 raft_consensus.cc:3037] T e01af2908eae46938a66d9db4dcb54e4 P 4bcacb667ac14937b041f7d4694b907b [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:34:19.838953 25963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e01af2908eae46938a66d9db4dcb54e4" candidate_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c33ed31d8faf4bc89a305efea7fa55a9"
I20260810 22:34:19.839089 25963 raft_consensus.cc:3037] T e01af2908eae46938a66d9db4dcb54e4 P c33ed31d8faf4bc89a305efea7fa55a9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:34:19.840281 26123 raft_consensus.cc:2454] T e01af2908eae46938a66d9db4dcb54e4 P 4bcacb667ac14937b041f7d4694b907b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd1ece4b27ae4e6c9c715a0045bac33c in term 1.
I20260810 22:34:19.840528 25963 raft_consensus.cc:2454] T e01af2908eae46938a66d9db4dcb54e4 P c33ed31d8faf4bc89a305efea7fa55a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd1ece4b27ae4e6c9c715a0045bac33c in term 1.
I20260810 22:34:19.840632 25740 leader_election.cc:302] T e01af2908eae46938a66d9db4dcb54e4 P bd1ece4b27ae4e6c9c715a0045bac33c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bcacb667ac14937b041f7d4694b907b, bd1ece4b27ae4e6c9c715a0045bac33c; no voters:
I20260810 22:34:19.840901 26214 raft_consensus.cc:2781] T e01af2908eae46938a66d9db4dcb54e4 P bd1ece4b27ae4e6c9c715a0045bac33c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:34:19.841070 26214 raft_consensus.cc:686] T e01af2908eae46938a66d9db4dcb54e4 P bd1ece4b27ae4e6c9c715a0045bac33c [term 1 LEADER]: Becoming Leader. State: Replica: bd1ece4b27ae4e6c9c715a0045bac33c, State: Running, Role: LEADER
I20260810 22:34:19.841444 26214 consensus_queue.cc:237] T e01af2908eae46938a66d9db4dcb54e4 P bd1ece4b27ae4e6c9c715a0045bac33c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bcacb667ac14937b041f7d4694b907b" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 41239 } } peers { permanent_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39183 } } peers { permanent_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33405 } }
I20260810 22:34:19.845718 25637 catalog_manager.cc:5495] T e01af2908eae46938a66d9db4dcb54e4 P bd1ece4b27ae4e6c9c715a0045bac33c reported cstate change: term changed from 0 to 1, leader changed from <none> to bd1ece4b27ae4e6c9c715a0045bac33c (127.16.233.65). New cstate: current_term: 1 leader_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bcacb667ac14937b041f7d4694b907b" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 41239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33405 } health_report { overall_health: HEALTHY } } }
I20260810 22:34:19.862129 26205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:19.857107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57162 (local address 127.16.233.65:33405)
0810 22:34:19.857400 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:19.857404 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:19.857447 (+ 43us) server_negotiation.cc:407] Connection header received
0810 22:34:19.857543 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:19.857545 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:19.857610 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:19.857714 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:34:19.858175 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:19.858890 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:19.860031 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:19.860318 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:19.860431 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:19.860950 (+ 519us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:34:19.861463 (+ 513us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:34:19.861567 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:19.861765 (+ 198us) server_negotiation.cc:299] Negotiation successful
0810 22:34:19.861911 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171}
I20260810 22:34:19.869988 26123 raft_consensus.cc:1261] T e01af2908eae46938a66d9db4dcb54e4 P 4bcacb667ac14937b041f7d4694b907b [term 1 FOLLOWER]: Refusing update from remote peer bd1ece4b27ae4e6c9c715a0045bac33c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:34:19.870507 25963 raft_consensus.cc:1261] T e01af2908eae46938a66d9db4dcb54e4 P c33ed31d8faf4bc89a305efea7fa55a9 [term 1 FOLLOWER]: Refusing update from remote peer bd1ece4b27ae4e6c9c715a0045bac33c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:34:19.870885 26214 consensus_queue.cc:1035] T e01af2908eae46938a66d9db4dcb54e4 P bd1ece4b27ae4e6c9c715a0045bac33c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bcacb667ac14937b041f7d4694b907b" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 41239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:34:19.871209 26212 consensus_queue.cc:1035] T e01af2908eae46938a66d9db4dcb54e4 P bd1ece4b27ae4e6c9c715a0045bac33c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:34:19.881516 26223 mvcc.cc:205] Tried to move back new op lower bound from 7317099560417406976 to 7317099560313786368. Current Snapshot: MvccSnapshot[applied={T|T < 7317099560417406976}]
I20260810 22:34:19.884008 26224 mvcc.cc:205] Tried to move back new op lower bound from 7317099560417406976 to 7317099560313786368. Current Snapshot: MvccSnapshot[applied={T|T < 7317099560417406976}]
I20260810 22:34:19.890606 25637 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45452:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:34:19.890872 25637 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45452:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:34:19.894233 25637 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 98283e123558435d982c4c04a9e1d6a0: Sending DeleteTablet for 3 replicas of tablet e01af2908eae46938a66d9db4dcb54e4
I20260810 22:34:19.895330 25941 tablet_service.cc:1514] Processing DeleteTablet for tablet e01af2908eae46938a66d9db4dcb54e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:34:19 UTC) from {username='oryx', principal='oryx/127.16.233.126@KRBTEST.COM'} at 127.0.0.1:34228
I20260810 22:34:19.895345 26101 tablet_service.cc:1514] Processing DeleteTablet for tablet e01af2908eae46938a66d9db4dcb54e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:34:19 UTC) from {username='oryx', principal='oryx/127.16.233.126@KRBTEST.COM'} at 127.0.0.1:56890
I20260810 22:34:19.895409 25793 tablet_service.cc:1514] Processing DeleteTablet for tablet e01af2908eae46938a66d9db4dcb54e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:34:19 UTC) from {username='oryx', principal='oryx/127.16.233.126@KRBTEST.COM'} at 127.0.0.1:57154
I20260810 22:34:19.896045 26233 tablet_replica.cc:330] T e01af2908eae46938a66d9db4dcb54e4 P bd1ece4b27ae4e6c9c715a0045bac33c: stopping tablet replica
I20260810 22:34:19.896174 26234 tablet_replica.cc:330] T e01af2908eae46938a66d9db4dcb54e4 P 4bcacb667ac14937b041f7d4694b907b: stopping tablet replica
I20260810 22:34:19.896478 26233 raft_consensus.cc:2229] T e01af2908eae46938a66d9db4dcb54e4 P bd1ece4b27ae4e6c9c715a0045bac33c [term 1 LEADER]: Raft consensus shutting down.
I20260810 22:34:19.896526 26234 raft_consensus.cc:2229] T e01af2908eae46938a66d9db4dcb54e4 P 4bcacb667ac14937b041f7d4694b907b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:34:19.896981 26234 raft_consensus.cc:2258] T e01af2908eae46938a66d9db4dcb54e4 P 4bcacb667ac14937b041f7d4694b907b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:34:19.897132 26233 raft_consensus.cc:2258] T e01af2908eae46938a66d9db4dcb54e4 P bd1ece4b27ae4e6c9c715a0045bac33c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:34:19.897258 25637 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45452:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 22:34:19.897815 25637 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:34:19.899045 26233 ts_tablet_manager.cc:1905] T e01af2908eae46938a66d9db4dcb54e4 P bd1ece4b27ae4e6c9c715a0045bac33c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:34:19.899051 26234 ts_tablet_manager.cc:1905] T e01af2908eae46938a66d9db4dcb54e4 P 4bcacb667ac14937b041f7d4694b907b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:34:19.899731 26232 tablet_replica.cc:330] T e01af2908eae46938a66d9db4dcb54e4 P c33ed31d8faf4bc89a305efea7fa55a9: stopping tablet replica
I20260810 22:34:19.900159 26232 raft_consensus.cc:2229] T e01af2908eae46938a66d9db4dcb54e4 P c33ed31d8faf4bc89a305efea7fa55a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:34:19.900549 26232 raft_consensus.cc:2258] T e01af2908eae46938a66d9db4dcb54e4 P c33ed31d8faf4bc89a305efea7fa55a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:34:19.903553 26233 ts_tablet_manager.cc:1918] T e01af2908eae46938a66d9db4dcb54e4 P bd1ece4b27ae4e6c9c715a0045bac33c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:34:19.903793 26233 log.cc:1196] T e01af2908eae46938a66d9db4dcb54e4 P bd1ece4b27ae4e6c9c715a0045bac33c: Deleting WAL directory at /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/ts-0/wal/wals/e01af2908eae46938a66d9db4dcb54e4
I20260810 22:34:19.904277 26233 ts_tablet_manager.cc:1939] T e01af2908eae46938a66d9db4dcb54e4 P bd1ece4b27ae4e6c9c715a0045bac33c: Deleting consensus metadata
I20260810 22:34:19.905061 26101 tablet_service.cc:1467] Processing CreateTablet for tablet 69b2cef8db6148a5b2317c81fe95f0a1 (DEFAULT_TABLE table=test-table [id=709c494f628f493baa19db2d61e3152b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:34:19.905503 26101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69b2cef8db6148a5b2317c81fe95f0a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:34:19.906812 26232 ts_tablet_manager.cc:1905] T e01af2908eae46938a66d9db4dcb54e4 P c33ed31d8faf4bc89a305efea7fa55a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:34:19.908427 26234 ts_tablet_manager.cc:1918] T e01af2908eae46938a66d9db4dcb54e4 P 4bcacb667ac14937b041f7d4694b907b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:34:19.908653 26234 log.cc:1196] T e01af2908eae46938a66d9db4dcb54e4 P 4bcacb667ac14937b041f7d4694b907b: Deleting WAL directory at /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/ts-2/wal/wals/e01af2908eae46938a66d9db4dcb54e4
I20260810 22:34:19.908684 26208 tablet_bootstrap.cc:492] T 69b2cef8db6148a5b2317c81fe95f0a1 P 4bcacb667ac14937b041f7d4694b907b: Bootstrap starting.
I20260810 22:34:19.909452 25620 catalog_manager.cc:4841] TS bd1ece4b27ae4e6c9c715a0045bac33c (127.16.233.65:33405): tablet e01af2908eae46938a66d9db4dcb54e4 (table test-table [id=01597677cdc94000ada79dd30c5986ab]) successfully deleted
I20260810 22:34:19.909946 26208 tablet_bootstrap.cc:654] T 69b2cef8db6148a5b2317c81fe95f0a1 P 4bcacb667ac14937b041f7d4694b907b: Neither blocks nor log segments found. Creating new log.
I20260810 22:34:19.910295 25793 tablet_service.cc:1467] Processing CreateTablet for tablet 69b2cef8db6148a5b2317c81fe95f0a1 (DEFAULT_TABLE table=test-table [id=709c494f628f493baa19db2d61e3152b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:34:19.910642 25793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69b2cef8db6148a5b2317c81fe95f0a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:34:19.910795 25941 tablet_service.cc:1467] Processing CreateTablet for tablet 69b2cef8db6148a5b2317c81fe95f0a1 (DEFAULT_TABLE table=test-table [id=709c494f628f493baa19db2d61e3152b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:34:19.911098 25941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69b2cef8db6148a5b2317c81fe95f0a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:34:19.913707 26207 tablet_bootstrap.cc:492] T 69b2cef8db6148a5b2317c81fe95f0a1 P c33ed31d8faf4bc89a305efea7fa55a9: Bootstrap starting.
I20260810 22:34:19.913707 26206 tablet_bootstrap.cc:492] T 69b2cef8db6148a5b2317c81fe95f0a1 P bd1ece4b27ae4e6c9c715a0045bac33c: Bootstrap starting.
I20260810 22:34:19.914569 26234 ts_tablet_manager.cc:1939] T e01af2908eae46938a66d9db4dcb54e4 P 4bcacb667ac14937b041f7d4694b907b: Deleting consensus metadata
I20260810 22:34:19.914939 26207 tablet_bootstrap.cc:654] T 69b2cef8db6148a5b2317c81fe95f0a1 P c33ed31d8faf4bc89a305efea7fa55a9: Neither blocks nor log segments found. Creating new log.
I20260810 22:34:19.914963 26206 tablet_bootstrap.cc:654] T 69b2cef8db6148a5b2317c81fe95f0a1 P bd1ece4b27ae4e6c9c715a0045bac33c: Neither blocks nor log segments found. Creating new log.
I20260810 22:34:19.915571 26232 ts_tablet_manager.cc:1918] T e01af2908eae46938a66d9db4dcb54e4 P c33ed31d8faf4bc89a305efea7fa55a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:34:19.915727 26232 log.cc:1196] T e01af2908eae46938a66d9db4dcb54e4 P c33ed31d8faf4bc89a305efea7fa55a9: Deleting WAL directory at /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/ts-1/wal/wals/e01af2908eae46938a66d9db4dcb54e4
I20260810 22:34:19.916146 26232 ts_tablet_manager.cc:1939] T e01af2908eae46938a66d9db4dcb54e4 P c33ed31d8faf4bc89a305efea7fa55a9: Deleting consensus metadata
I20260810 22:34:19.916621 25621 catalog_manager.cc:4841] TS 4bcacb667ac14937b041f7d4694b907b (127.16.233.67:41239): tablet e01af2908eae46938a66d9db4dcb54e4 (table test-table [id=01597677cdc94000ada79dd30c5986ab]) successfully deleted
I20260810 22:34:19.917582 25620 catalog_manager.cc:4841] TS c33ed31d8faf4bc89a305efea7fa55a9 (127.16.233.66:39183): tablet e01af2908eae46938a66d9db4dcb54e4 (table test-table [id=01597677cdc94000ada79dd30c5986ab]) successfully deleted
I20260810 22:34:19.917706 26208 tablet_bootstrap.cc:492] T 69b2cef8db6148a5b2317c81fe95f0a1 P 4bcacb667ac14937b041f7d4694b907b: No bootstrap required, opened a new log
I20260810 22:34:19.917819 26208 ts_tablet_manager.cc:1397] T 69b2cef8db6148a5b2317c81fe95f0a1 P 4bcacb667ac14937b041f7d4694b907b: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260810 22:34:19.918385 26207 tablet_bootstrap.cc:492] T 69b2cef8db6148a5b2317c81fe95f0a1 P c33ed31d8faf4bc89a305efea7fa55a9: No bootstrap required, opened a new log
I20260810 22:34:19.918493 26207 ts_tablet_manager.cc:1397] T 69b2cef8db6148a5b2317c81fe95f0a1 P c33ed31d8faf4bc89a305efea7fa55a9: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260810 22:34:19.918807 26208 raft_consensus.cc:348] T 69b2cef8db6148a5b2317c81fe95f0a1 P 4bcacb667ac14937b041f7d4694b907b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33405 } } peers { permanent_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39183 } } peers { permanent_uuid: "4bcacb667ac14937b041f7d4694b907b" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 41239 } }
I20260810 22:34:19.918982 26208 raft_consensus.cc:374] T 69b2cef8db6148a5b2317c81fe95f0a1 P 4bcacb667ac14937b041f7d4694b907b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:34:19.918969 26207 raft_consensus.cc:348] T 69b2cef8db6148a5b2317c81fe95f0a1 P c33ed31d8faf4bc89a305efea7fa55a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33405 } } peers { permanent_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39183 } } peers { permanent_uuid: "4bcacb667ac14937b041f7d4694b907b" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 41239 } }
I20260810 22:34:19.919035 26208 raft_consensus.cc:729] T 69b2cef8db6148a5b2317c81fe95f0a1 P 4bcacb667ac14937b041f7d4694b907b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bcacb667ac14937b041f7d4694b907b, State: Initialized, Role: FOLLOWER
I20260810 22:34:19.919096 26207 raft_consensus.cc:374] T 69b2cef8db6148a5b2317c81fe95f0a1 P c33ed31d8faf4bc89a305efea7fa55a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:34:19.919130 26207 raft_consensus.cc:729] T 69b2cef8db6148a5b2317c81fe95f0a1 P c33ed31d8faf4bc89a305efea7fa55a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c33ed31d8faf4bc89a305efea7fa55a9, State: Initialized, Role: FOLLOWER
I20260810 22:34:19.919226 26208 consensus_queue.cc:260] T 69b2cef8db6148a5b2317c81fe95f0a1 P 4bcacb667ac14937b041f7d4694b907b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33405 } } peers { permanent_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39183 } } peers { permanent_uuid: "4bcacb667ac14937b041f7d4694b907b" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 41239 } }
I20260810 22:34:19.919258 26207 consensus_queue.cc:260] T 69b2cef8db6148a5b2317c81fe95f0a1 P c33ed31d8faf4bc89a305efea7fa55a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33405 } } peers { permanent_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39183 } } peers { permanent_uuid: "4bcacb667ac14937b041f7d4694b907b" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 41239 } }
I20260810 22:34:19.919540 26207 ts_tablet_manager.cc:1428] T 69b2cef8db6148a5b2317c81fe95f0a1 P c33ed31d8faf4bc89a305efea7fa55a9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 22:34:19.919590 26208 ts_tablet_manager.cc:1428] T 69b2cef8db6148a5b2317c81fe95f0a1 P 4bcacb667ac14937b041f7d4694b907b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260810 22:34:19.920783 26206 tablet_bootstrap.cc:492] T 69b2cef8db6148a5b2317c81fe95f0a1 P bd1ece4b27ae4e6c9c715a0045bac33c: No bootstrap required, opened a new log
I20260810 22:34:19.920986 26206 ts_tablet_manager.cc:1397] T 69b2cef8db6148a5b2317c81fe95f0a1 P bd1ece4b27ae4e6c9c715a0045bac33c: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.001s
I20260810 22:34:19.921511 26206 raft_consensus.cc:348] T 69b2cef8db6148a5b2317c81fe95f0a1 P bd1ece4b27ae4e6c9c715a0045bac33c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33405 } } peers { permanent_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39183 } } peers { permanent_uuid: "4bcacb667ac14937b041f7d4694b907b" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 41239 } }
I20260810 22:34:19.921649 26206 raft_consensus.cc:374] T 69b2cef8db6148a5b2317c81fe95f0a1 P bd1ece4b27ae4e6c9c715a0045bac33c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:34:19.921684 26206 raft_consensus.cc:729] T 69b2cef8db6148a5b2317c81fe95f0a1 P bd1ece4b27ae4e6c9c715a0045bac33c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd1ece4b27ae4e6c9c715a0045bac33c, State: Initialized, Role: FOLLOWER
I20260810 22:34:19.921866 26206 consensus_queue.cc:260] T 69b2cef8db6148a5b2317c81fe95f0a1 P bd1ece4b27ae4e6c9c715a0045bac33c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33405 } } peers { permanent_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39183 } } peers { permanent_uuid: "4bcacb667ac14937b041f7d4694b907b" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 41239 } }
I20260810 22:34:19.922170 26206 ts_tablet_manager.cc:1428] T 69b2cef8db6148a5b2317c81fe95f0a1 P bd1ece4b27ae4e6c9c715a0045bac33c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
W20260810 22:34:19.929491 25869 tablet.cc:2369] T 69b2cef8db6148a5b2317c81fe95f0a1 P bd1ece4b27ae4e6c9c715a0045bac33c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 22:34:19.947350 26021 tablet.cc:2369] T 69b2cef8db6148a5b2317c81fe95f0a1 P c33ed31d8faf4bc89a305efea7fa55a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 22:34:19.955232 26178 tablet.cc:2369] T 69b2cef8db6148a5b2317c81fe95f0a1 P 4bcacb667ac14937b041f7d4694b907b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:34:19.999583 26213 raft_consensus.cc:482] T 69b2cef8db6148a5b2317c81fe95f0a1 P 4bcacb667ac14937b041f7d4694b907b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:34:19.999886 26213 raft_consensus.cc:504] T 69b2cef8db6148a5b2317c81fe95f0a1 P 4bcacb667ac14937b041f7d4694b907b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33405 } } peers { permanent_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39183 } } peers { permanent_uuid: "4bcacb667ac14937b041f7d4694b907b" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 41239 } }
I20260810 22:34:20.001397 26213 leader_election.cc:288] T 69b2cef8db6148a5b2317c81fe95f0a1 P 4bcacb667ac14937b041f7d4694b907b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd1ece4b27ae4e6c9c715a0045bac33c (127.16.233.65:33405), c33ed31d8faf4bc89a305efea7fa55a9 (127.16.233.66:39183)
I20260810 22:34:20.006595 26205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:20.001853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:36425 (local address 127.16.233.65:33405)
0810 22:34:20.002006 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:20.002011 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:20.002030 (+ 19us) server_negotiation.cc:407] Connection header received
0810 22:34:20.002221 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:20.002224 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:20.002327 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:20.002420 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:34:20.002968 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:20.003830 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:20.005259 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:20.005871 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:20.006016 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:20.006080 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:20.006352 (+ 272us) server_negotiation.cc:299] Negotiation successful
0810 22:34:20.006471 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 22:34:20.006603 26054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:20.001723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.65:33405 (local address 127.16.233.67:36425)
0810 22:34:20.001920 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:20.001941 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:20.002069 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:20.002572 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:20.002575 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:20.002628 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:34:20.002801 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:20.002810 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:20.004000 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:20.004005 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:20.005096 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:20.005107 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:20.006040 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:20.006045 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:20.006178 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:20.006208 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 22:34:20.006304 (+ 96us) client_negotiation.cc:240] Negotiation successful
0810 22:34:20.006404 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"mutex_wait_us":61,"spinlock_wait_cycles":2688}
I20260810 22:34:20.007499 26238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:20.002037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.66:39183 (local address 127.16.233.67:35447)
0810 22:34:20.002462 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:20.002480 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:20.002633 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:20.003044 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:20.003047 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:20.003062 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:34:20.003164 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:20.003169 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:20.004038 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:20.004042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:20.005404 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:20.005418 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:20.006951 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:20.006955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:20.007062 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:20.007094 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 22:34:20.007216 (+ 122us) client_negotiation.cc:240] Negotiation successful
0810 22:34:20.007314 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"mutex_wait_us":3,"spinlock_wait_cycles":31872,"thread_start_us":119,"threads_started":1}
I20260810 22:34:20.007490 26202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:20.002420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:35447 (local address 127.16.233.66:39183)
0810 22:34:20.002568 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:20.002573 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:20.002589 (+ 16us) server_negotiation.cc:407] Connection header received
0810 22:34:20.002782 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:20.002786 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:20.002844 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:20.002930 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:34:20.003299 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:20.003854 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:20.005571 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:20.006221 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:20.006381 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:20.006432 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:20.007249 (+ 817us) server_negotiation.cc:299] Negotiation successful
0810 22:34:20.007355 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 22:34:20.007417 25816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69b2cef8db6148a5b2317c81fe95f0a1" candidate_uuid: "4bcacb667ac14937b041f7d4694b907b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" is_pre_election: true
I20260810 22:34:20.007719 25816 raft_consensus.cc:2454] T 69b2cef8db6148a5b2317c81fe95f0a1 P bd1ece4b27ae4e6c9c715a0045bac33c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bcacb667ac14937b041f7d4694b907b in term 0.
I20260810 22:34:20.008168 25971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69b2cef8db6148a5b2317c81fe95f0a1" candidate_uuid: "4bcacb667ac14937b041f7d4694b907b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" is_pre_election: true
I20260810 22:34:20.008284 26047 leader_election.cc:302] T 69b2cef8db6148a5b2317c81fe95f0a1 P 4bcacb667ac14937b041f7d4694b907b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bcacb667ac14937b041f7d4694b907b, bd1ece4b27ae4e6c9c715a0045bac33c; no voters:
I20260810 22:34:20.008353 25971 raft_consensus.cc:2454] T 69b2cef8db6148a5b2317c81fe95f0a1 P c33ed31d8faf4bc89a305efea7fa55a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bcacb667ac14937b041f7d4694b907b in term 0.
I20260810 22:34:20.008675 26213 raft_consensus.cc:2781] T 69b2cef8db6148a5b2317c81fe95f0a1 P 4bcacb667ac14937b041f7d4694b907b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:34:20.008747 26213 raft_consensus.cc:482] T 69b2cef8db6148a5b2317c81fe95f0a1 P 4bcacb667ac14937b041f7d4694b907b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:34:20.008800 26213 raft_consensus.cc:3037] T 69b2cef8db6148a5b2317c81fe95f0a1 P 4bcacb667ac14937b041f7d4694b907b [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:34:20.010111 26213 raft_consensus.cc:504] T 69b2cef8db6148a5b2317c81fe95f0a1 P 4bcacb667ac14937b041f7d4694b907b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33405 } } peers { permanent_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39183 } } peers { permanent_uuid: "4bcacb667ac14937b041f7d4694b907b" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 41239 } }
I20260810 22:34:20.010569 26213 leader_election.cc:288] T 69b2cef8db6148a5b2317c81fe95f0a1 P 4bcacb667ac14937b041f7d4694b907b [CANDIDATE]: Term 1 election: Requested vote from peers bd1ece4b27ae4e6c9c715a0045bac33c (127.16.233.65:33405), c33ed31d8faf4bc89a305efea7fa55a9 (127.16.233.66:39183)
I20260810 22:34:20.011157 25971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69b2cef8db6148a5b2317c81fe95f0a1" candidate_uuid: "4bcacb667ac14937b041f7d4694b907b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c33ed31d8faf4bc89a305efea7fa55a9"
I20260810 22:34:20.011240 25816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69b2cef8db6148a5b2317c81fe95f0a1" candidate_uuid: "4bcacb667ac14937b041f7d4694b907b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c"
I20260810 22:34:20.011317 25971 raft_consensus.cc:3037] T 69b2cef8db6148a5b2317c81fe95f0a1 P c33ed31d8faf4bc89a305efea7fa55a9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:34:20.011385 25816 raft_consensus.cc:3037] T 69b2cef8db6148a5b2317c81fe95f0a1 P bd1ece4b27ae4e6c9c715a0045bac33c [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:34:20.012748 25971 raft_consensus.cc:2454] T 69b2cef8db6148a5b2317c81fe95f0a1 P c33ed31d8faf4bc89a305efea7fa55a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bcacb667ac14937b041f7d4694b907b in term 1.
I20260810 22:34:20.012739 25816 raft_consensus.cc:2454] T 69b2cef8db6148a5b2317c81fe95f0a1 P bd1ece4b27ae4e6c9c715a0045bac33c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bcacb667ac14937b041f7d4694b907b in term 1.
I20260810 22:34:20.013188 26047 leader_election.cc:302] T 69b2cef8db6148a5b2317c81fe95f0a1 P 4bcacb667ac14937b041f7d4694b907b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bcacb667ac14937b041f7d4694b907b, bd1ece4b27ae4e6c9c715a0045bac33c; no voters:
I20260810 22:34:20.013451 26213 raft_consensus.cc:2781] T 69b2cef8db6148a5b2317c81fe95f0a1 P 4bcacb667ac14937b041f7d4694b907b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:34:20.013760 26213 raft_consensus.cc:686] T 69b2cef8db6148a5b2317c81fe95f0a1 P 4bcacb667ac14937b041f7d4694b907b [term 1 LEADER]: Becoming Leader. State: Replica: 4bcacb667ac14937b041f7d4694b907b, State: Running, Role: LEADER
I20260810 22:34:20.014084 26213 consensus_queue.cc:237] T 69b2cef8db6148a5b2317c81fe95f0a1 P 4bcacb667ac14937b041f7d4694b907b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33405 } } peers { permanent_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39183 } } peers { permanent_uuid: "4bcacb667ac14937b041f7d4694b907b" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 41239 } }
I20260810 22:34:20.018010 25636 catalog_manager.cc:5495] T 69b2cef8db6148a5b2317c81fe95f0a1 P 4bcacb667ac14937b041f7d4694b907b reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bcacb667ac14937b041f7d4694b907b (127.16.233.67). New cstate: current_term: 1 leader_uuid: "4bcacb667ac14937b041f7d4694b907b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bcacb667ac14937b041f7d4694b907b" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 41239 } health_report { overall_health: HEALTHY } } }
Aug 10 22:34:20 dist-test-slave-77v9 krb5kdc[25588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401258, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.126@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM
I20260810 22:34:20.078727 26249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:20.066899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:45483 (local address 127.0.0.1:45462)
0810 22:34:20.067250 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:20.067268 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:20.067408 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:20.067728 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:20.067731 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:20.068227 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:20.068341 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:20.068348 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:20.069614 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:20.069618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:20.070293 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:20.070304 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:20.070971 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:20.070974 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:20.071053 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:20.071897 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:20.071912 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:20.073831 (+ 1919us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:34:20.076514 (+ 2683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:20.076526 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:20.076532 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:20.077000 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:20.077413 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:20.077416 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:20.077417 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:20.077536 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:20.078124 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:34:20.078130 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:34:20.078269 (+ 139us) client_negotiation.cc:769] Sending connection context
0810 22:34:20.078388 (+ 119us) client_negotiation.cc:240] Negotiation successful
0810 22:34:20.078568 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":128,"threads_started":1}
I20260810 22:34:20.078881 25695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:20.067038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45462 (local address 127.16.233.126:45483)
0810 22:34:20.067219 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:20.067225 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:20.067375 (+ 150us) server_negotiation.cc:407] Connection header received
0810 22:34:20.067511 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:20.067514 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:20.067568 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:20.067675 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:20.068551 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:20.069453 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:20.070462 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:20.070825 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:20.070940 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:20.074081 (+ 3141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:20.074085 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:20.074088 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:20.074126 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:20.076316 (+ 2190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:20.077118 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:20.077123 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:20.077125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:20.077269 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:20.077673 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:20.077676 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:20.077678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:20.077972 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:20.078092 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:20.078469 (+ 377us) server_negotiation.cc:299] Negotiation successful
0810 22:34:20.078666 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":76}
I20260810 22:34:20.084188 25636 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.16.233.126@KRBTEST.COM'} at 127.0.0.1:45462:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 22:34:20.085279 25636 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:34:20.091827 25793 tablet_service.cc:1467] Processing CreateTablet for tablet 3ff9dc9e14bb40bbb6d8aad7de5453d3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0d1891f066bf4248a8e3d455d220a100]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 22:34:20.091827 25941 tablet_service.cc:1467] Processing CreateTablet for tablet 3ff9dc9e14bb40bbb6d8aad7de5453d3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0d1891f066bf4248a8e3d455d220a100]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 22:34:20.092051 26101 tablet_service.cc:1467] Processing CreateTablet for tablet 3ff9dc9e14bb40bbb6d8aad7de5453d3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0d1891f066bf4248a8e3d455d220a100]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 22:34:20.092322 25941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ff9dc9e14bb40bbb6d8aad7de5453d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:34:20.092329 25793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ff9dc9e14bb40bbb6d8aad7de5453d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:34:20.092519 26101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ff9dc9e14bb40bbb6d8aad7de5453d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:34:20.095453 26207 tablet_bootstrap.cc:492] T 3ff9dc9e14bb40bbb6d8aad7de5453d3 P c33ed31d8faf4bc89a305efea7fa55a9: Bootstrap starting.
I20260810 22:34:20.095463 26206 tablet_bootstrap.cc:492] T 3ff9dc9e14bb40bbb6d8aad7de5453d3 P bd1ece4b27ae4e6c9c715a0045bac33c: Bootstrap starting.
I20260810 22:34:20.095556 26208 tablet_bootstrap.cc:492] T 3ff9dc9e14bb40bbb6d8aad7de5453d3 P 4bcacb667ac14937b041f7d4694b907b: Bootstrap starting.
I20260810 22:34:20.096392 26207 tablet_bootstrap.cc:654] T 3ff9dc9e14bb40bbb6d8aad7de5453d3 P c33ed31d8faf4bc89a305efea7fa55a9: Neither blocks nor log segments found. Creating new log.
I20260810 22:34:20.096614 26206 tablet_bootstrap.cc:654] T 3ff9dc9e14bb40bbb6d8aad7de5453d3 P bd1ece4b27ae4e6c9c715a0045bac33c: Neither blocks nor log segments found. Creating new log.
I20260810 22:34:20.096614 26208 tablet_bootstrap.cc:654] T 3ff9dc9e14bb40bbb6d8aad7de5453d3 P 4bcacb667ac14937b041f7d4694b907b: Neither blocks nor log segments found. Creating new log.
I20260810 22:34:20.097689 26207 tablet_bootstrap.cc:492] T 3ff9dc9e14bb40bbb6d8aad7de5453d3 P c33ed31d8faf4bc89a305efea7fa55a9: No bootstrap required, opened a new log
I20260810 22:34:20.097797 26207 ts_tablet_manager.cc:1397] T 3ff9dc9e14bb40bbb6d8aad7de5453d3 P c33ed31d8faf4bc89a305efea7fa55a9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260810 22:34:20.098342 26207 raft_consensus.cc:348] T 3ff9dc9e14bb40bbb6d8aad7de5453d3 P c33ed31d8faf4bc89a305efea7fa55a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33405 } } peers { permanent_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39183 } } peers { permanent_uuid: "4bcacb667ac14937b041f7d4694b907b" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 41239 } }
I20260810 22:34:20.098479 26207 raft_consensus.cc:374] T 3ff9dc9e14bb40bbb6d8aad7de5453d3 P c33ed31d8faf4bc89a305efea7fa55a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:34:20.098522 26207 raft_consensus.cc:729] T 3ff9dc9e14bb40bbb6d8aad7de5453d3 P c33ed31d8faf4bc89a305efea7fa55a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c33ed31d8faf4bc89a305efea7fa55a9, State: Initialized, Role: FOLLOWER
I20260810 22:34:20.098680 26208 tablet_bootstrap.cc:492] T 3ff9dc9e14bb40bbb6d8aad7de5453d3 P 4bcacb667ac14937b041f7d4694b907b: No bootstrap required, opened a new log
I20260810 22:34:20.098657 26207 consensus_queue.cc:260] T 3ff9dc9e14bb40bbb6d8aad7de5453d3 P c33ed31d8faf4bc89a305efea7fa55a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33405 } } peers { permanent_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39183 } } peers { permanent_uuid: "4bcacb667ac14937b041f7d4694b907b" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 41239 } }
I20260810 22:34:20.098802 26208 ts_tablet_manager.cc:1397] T 3ff9dc9e14bb40bbb6d8aad7de5453d3 P 4bcacb667ac14937b041f7d4694b907b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 22:34:20.098974 26207 ts_tablet_manager.cc:1428] T 3ff9dc9e14bb40bbb6d8aad7de5453d3 P c33ed31d8faf4bc89a305efea7fa55a9: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 22:34:20.099012 26206 tablet_bootstrap.cc:492] T 3ff9dc9e14bb40bbb6d8aad7de5453d3 P bd1ece4b27ae4e6c9c715a0045bac33c: No bootstrap required, opened a new log
I20260810 22:34:20.099102 26206 ts_tablet_manager.cc:1397] T 3ff9dc9e14bb40bbb6d8aad7de5453d3 P bd1ece4b27ae4e6c9c715a0045bac33c: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260810 22:34:20.099203 26218 tablet_replica.cc:439] T 3ff9dc9e14bb40bbb6d8aad7de5453d3 P c33ed31d8faf4bc89a305efea7fa55a9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33405 } } peers { permanent_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39183 } } peers { permanent_uuid: "4bcacb667ac14937b041f7d4694b907b" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 41239 } } }
I20260810 22:34:20.099385 26218 tablet_replica.cc:442] T 3ff9dc9e14bb40bbb6d8aad7de5453d3 P c33ed31d8faf4bc89a305efea7fa55a9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 22:34:20.099385 26208 raft_consensus.cc:348] T 3ff9dc9e14bb40bbb6d8aad7de5453d3 P 4bcacb667ac14937b041f7d4694b907b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33405 } } peers { permanent_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39183 } } peers { permanent_uuid: "4bcacb667ac14937b041f7d4694b907b" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 41239 } }
I20260810 22:34:20.099536 26208 raft_consensus.cc:374] T 3ff9dc9e14bb40bbb6d8aad7de5453d3 P 4bcacb667ac14937b041f7d4694b907b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:34:20.099589 26208 raft_consensus.cc:729] T 3ff9dc9e14bb40bbb6d8aad7de5453d3 P 4bcacb667ac14937b041f7d4694b907b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bcacb667ac14937b041f7d4694b907b, State: Initialized, Role: FOLLOWER
I20260810 22:34:20.099596 26206 raft_consensus.cc:348] T 3ff9dc9e14bb40bbb6d8aad7de5453d3 P bd1ece4b27ae4e6c9c715a0045bac33c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33405 } } peers { permanent_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39183 } } peers { permanent_uuid: "4bcacb667ac14937b041f7d4694b907b" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 41239 } }
I20260810 22:34:20.099745 26206 raft_consensus.cc:374] T 3ff9dc9e14bb40bbb6d8aad7de5453d3 P bd1ece4b27ae4e6c9c715a0045bac33c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:34:20.099792 26206 raft_consensus.cc:729] T 3ff9dc9e14bb40bbb6d8aad7de5453d3 P bd1ece4b27ae4e6c9c715a0045bac33c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd1ece4b27ae4e6c9c715a0045bac33c, State: Initialized, Role: FOLLOWER
I20260810 22:34:20.099762 26208 consensus_queue.cc:260] T 3ff9dc9e14bb40bbb6d8aad7de5453d3 P 4bcacb667ac14937b041f7d4694b907b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33405 } } peers { permanent_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39183 } } peers { permanent_uuid: "4bcacb667ac14937b041f7d4694b907b" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 41239 } }
I20260810 22:34:20.099947 26206 consensus_queue.cc:260] T 3ff9dc9e14bb40bbb6d8aad7de5453d3 P bd1ece4b27ae4e6c9c715a0045bac33c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33405 } } peers { permanent_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39183 } } peers { permanent_uuid: "4bcacb667ac14937b041f7d4694b907b" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 41239 } }
I20260810 22:34:20.100111 26208 ts_tablet_manager.cc:1428] T 3ff9dc9e14bb40bbb6d8aad7de5453d3 P 4bcacb667ac14937b041f7d4694b907b: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260810 22:34:20.100277 26206 ts_tablet_manager.cc:1428] T 3ff9dc9e14bb40bbb6d8aad7de5453d3 P bd1ece4b27ae4e6c9c715a0045bac33c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 22:34:20.100245 26228 tablet_replica.cc:439] T 3ff9dc9e14bb40bbb6d8aad7de5453d3 P bd1ece4b27ae4e6c9c715a0045bac33c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33405 } } peers { permanent_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39183 } } peers { permanent_uuid: "4bcacb667ac14937b041f7d4694b907b" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 41239 } } }
I20260810 22:34:20.100430 26228 tablet_replica.cc:442] T 3ff9dc9e14bb40bbb6d8aad7de5453d3 P bd1ece4b27ae4e6c9c715a0045bac33c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 22:34:20.100395 26213 tablet_replica.cc:439] T 3ff9dc9e14bb40bbb6d8aad7de5453d3 P 4bcacb667ac14937b041f7d4694b907b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33405 } } peers { permanent_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39183 } } peers { permanent_uuid: "4bcacb667ac14937b041f7d4694b907b" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 41239 } } }
I20260810 22:34:20.100546 26213 tablet_replica.cc:442] T 3ff9dc9e14bb40bbb6d8aad7de5453d3 P 4bcacb667ac14937b041f7d4694b907b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 22:34:20.133503 26218 raft_consensus.cc:482] T 3ff9dc9e14bb40bbb6d8aad7de5453d3 P c33ed31d8faf4bc89a305efea7fa55a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:34:20.133728 26218 raft_consensus.cc:504] T 3ff9dc9e14bb40bbb6d8aad7de5453d3 P c33ed31d8faf4bc89a305efea7fa55a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33405 } } peers { permanent_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39183 } } peers { permanent_uuid: "4bcacb667ac14937b041f7d4694b907b" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 41239 } }
I20260810 22:34:20.135221 26218 leader_election.cc:288] T 3ff9dc9e14bb40bbb6d8aad7de5453d3 P c33ed31d8faf4bc89a305efea7fa55a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd1ece4b27ae4e6c9c715a0045bac33c (127.16.233.65:33405), 4bcacb667ac14937b041f7d4694b907b (127.16.233.67:41239)
I20260810 22:34:20.142118 26203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:20.135705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:53233 (local address 127.16.233.67:41239)
0810 22:34:20.135879 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:20.135885 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:20.136142 (+ 257us) server_negotiation.cc:407] Connection header received
0810 22:34:20.136271 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:20.136275 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:20.136334 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:20.136425 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:34:20.137275 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:20.138115 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:20.140293 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:20.140987 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:20.141165 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:20.141225 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:20.141860 (+ 635us) server_negotiation.cc:299] Negotiation successful
0810 22:34:20.141965 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 22:34:20.142118 26205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:20.135621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:44393 (local address 127.16.233.65:33405)
0810 22:34:20.135776 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:20.135783 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:20.136142 (+ 359us) server_negotiation.cc:407] Connection header received
0810 22:34:20.136234 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:20.136237 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:20.136301 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:20.136426 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:34:20.137277 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:20.138132 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:20.140293 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:20.141055 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:20.141174 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:20.141225 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:20.141857 (+ 632us) server_negotiation.cc:299] Negotiation successful
0810 22:34:20.141984 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 22:34:20.142141 26253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:20.135526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.65:33405 (local address 127.16.233.66:44393)
0810 22:34:20.135936 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:20.135954 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:20.136094 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:20.136488 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:20.136492 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:20.136546 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:34:20.137073 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:20.137083 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:20.138305 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:20.138310 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:20.140068 (+ 1758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:20.140084 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:20.141243 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:20.141248 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:20.141569 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:20.141640 (+ 71us) client_negotiation.cc:769] Sending connection context
0810 22:34:20.141760 (+ 120us) client_negotiation.cc:240] Negotiation successful
0810 22:34:20.141908 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":512,"spinlock_wait_cycles":8192,"thread_start_us":106,"threads_started":1}
I20260810 22:34:20.142150 26254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:20.135557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.67:41239 (local address 127.16.233.66:53233)
0810 22:34:20.135994 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:20.136010 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:20.136094 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:20.136488 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:20.136492 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:20.136545 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:34:20.137070 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:20.137083 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:20.138305 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:20.138310 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:20.140068 (+ 1758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:20.140084 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:20.141243 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:20.141248 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:20.141567 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:20.141643 (+ 76us) client_negotiation.cc:769] Sending connection context
0810 22:34:20.141760 (+ 117us) client_negotiation.cc:240] Negotiation successful
0810 22:34:20.141909 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"mutex_wait_us":243,"spinlock_wait_cycles":24192,"thread_start_us":102,"threads_started":1}
I20260810 22:34:20.142802 25816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff9dc9e14bb40bbb6d8aad7de5453d3" candidate_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" is_pre_election: true
I20260810 22:34:20.143038 25816 raft_consensus.cc:2454] T 3ff9dc9e14bb40bbb6d8aad7de5453d3 P bd1ece4b27ae4e6c9c715a0045bac33c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c33ed31d8faf4bc89a305efea7fa55a9 in term 0.
I20260810 22:34:20.143244 26123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff9dc9e14bb40bbb6d8aad7de5453d3" candidate_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bcacb667ac14937b041f7d4694b907b" is_pre_election: true
I20260810 22:34:20.143431 26123 raft_consensus.cc:2454] T 3ff9dc9e14bb40bbb6d8aad7de5453d3 P 4bcacb667ac14937b041f7d4694b907b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c33ed31d8faf4bc89a305efea7fa55a9 in term 0.
I20260810 22:34:20.143558 25894 leader_election.cc:302] T 3ff9dc9e14bb40bbb6d8aad7de5453d3 P c33ed31d8faf4bc89a305efea7fa55a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd1ece4b27ae4e6c9c715a0045bac33c, c33ed31d8faf4bc89a305efea7fa55a9; no voters:
I20260810 22:34:20.143905 26218 raft_consensus.cc:2781] T 3ff9dc9e14bb40bbb6d8aad7de5453d3 P c33ed31d8faf4bc89a305efea7fa55a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:34:20.144001 26218 raft_consensus.cc:482] T 3ff9dc9e14bb40bbb6d8aad7de5453d3 P c33ed31d8faf4bc89a305efea7fa55a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:34:20.144062 26218 raft_consensus.cc:3037] T 3ff9dc9e14bb40bbb6d8aad7de5453d3 P c33ed31d8faf4bc89a305efea7fa55a9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:34:20.145494 26218 raft_consensus.cc:504] T 3ff9dc9e14bb40bbb6d8aad7de5453d3 P c33ed31d8faf4bc89a305efea7fa55a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33405 } } peers { permanent_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39183 } } peers { permanent_uuid: "4bcacb667ac14937b041f7d4694b907b" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 41239 } }
I20260810 22:34:20.145962 26218 leader_election.cc:288] T 3ff9dc9e14bb40bbb6d8aad7de5453d3 P c33ed31d8faf4bc89a305efea7fa55a9 [CANDIDATE]: Term 1 election: Requested vote from peers bd1ece4b27ae4e6c9c715a0045bac33c (127.16.233.65:33405), 4bcacb667ac14937b041f7d4694b907b (127.16.233.67:41239)
I20260810 22:34:20.146507 25816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff9dc9e14bb40bbb6d8aad7de5453d3" candidate_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c"
I20260810 22:34:20.146507 26123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ff9dc9e14bb40bbb6d8aad7de5453d3" candidate_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bcacb667ac14937b041f7d4694b907b"
I20260810 22:34:20.146668 25816 raft_consensus.cc:3037] T 3ff9dc9e14bb40bbb6d8aad7de5453d3 P bd1ece4b27ae4e6c9c715a0045bac33c [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:34:20.146660 26123 raft_consensus.cc:3037] T 3ff9dc9e14bb40bbb6d8aad7de5453d3 P 4bcacb667ac14937b041f7d4694b907b [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:34:20.147984 26123 raft_consensus.cc:2454] T 3ff9dc9e14bb40bbb6d8aad7de5453d3 P 4bcacb667ac14937b041f7d4694b907b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c33ed31d8faf4bc89a305efea7fa55a9 in term 1.
I20260810 22:34:20.147984 25816 raft_consensus.cc:2454] T 3ff9dc9e14bb40bbb6d8aad7de5453d3 P bd1ece4b27ae4e6c9c715a0045bac33c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c33ed31d8faf4bc89a305efea7fa55a9 in term 1.
I20260810 22:34:20.148375 25894 leader_election.cc:302] T 3ff9dc9e14bb40bbb6d8aad7de5453d3 P c33ed31d8faf4bc89a305efea7fa55a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd1ece4b27ae4e6c9c715a0045bac33c, c33ed31d8faf4bc89a305efea7fa55a9; no voters:
I20260810 22:34:20.148594 26218 raft_consensus.cc:2781] T 3ff9dc9e14bb40bbb6d8aad7de5453d3 P c33ed31d8faf4bc89a305efea7fa55a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:34:20.148916 26218 raft_consensus.cc:686] T 3ff9dc9e14bb40bbb6d8aad7de5453d3 P c33ed31d8faf4bc89a305efea7fa55a9 [term 1 LEADER]: Becoming Leader. State: Replica: c33ed31d8faf4bc89a305efea7fa55a9, State: Running, Role: LEADER
I20260810 22:34:20.149192 26218 consensus_queue.cc:237] T 3ff9dc9e14bb40bbb6d8aad7de5453d3 P c33ed31d8faf4bc89a305efea7fa55a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33405 } } peers { permanent_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39183 } } peers { permanent_uuid: "4bcacb667ac14937b041f7d4694b907b" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 41239 } }
I20260810 22:34:20.150674 26255 tablet_replica.cc:439] T 3ff9dc9e14bb40bbb6d8aad7de5453d3 P c33ed31d8faf4bc89a305efea7fa55a9: TxnStatusTablet state changed. Reason: New leader c33ed31d8faf4bc89a305efea7fa55a9. Latest consensus state: current_term: 1 leader_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33405 } } peers { permanent_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39183 } } peers { permanent_uuid: "4bcacb667ac14937b041f7d4694b907b" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 41239 } } }
I20260810 22:34:20.150841 26255 tablet_replica.cc:442] T 3ff9dc9e14bb40bbb6d8aad7de5453d3 P c33ed31d8faf4bc89a305efea7fa55a9: This TxnStatusTablet replica's current role is: LEADER
I20260810 22:34:20.151170 26257 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 22:34:20.151341 26257 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 22:34:20.152168 25638 catalog_manager.cc:5495] T 3ff9dc9e14bb40bbb6d8aad7de5453d3 P c33ed31d8faf4bc89a305efea7fa55a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to c33ed31d8faf4bc89a305efea7fa55a9 (127.16.233.66). New cstate: current_term: 1 leader_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4bcacb667ac14937b041f7d4694b907b" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 41239 } health_report { overall_health: UNKNOWN } } }
I20260810 22:34:20.169585 26249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:20.165199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.66:39183 (local address 127.0.0.1:34240)
0810 22:34:20.165355 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:20.165374 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:20.165487 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:20.166195 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:20.166199 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:20.166208 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 22:34:20.166326 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:20.166339 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:20.167093 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:20.167097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:20.167729 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:20.167738 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:20.168302 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:20.168305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:20.168402 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:20.168451 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 22:34:20.169257 (+ 806us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 22:34:20.169276 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 22:34:20.169347 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 22:34:20.169448 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260810 22:34:20.169585 26202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:20.165770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34240 (local address 127.16.233.66:39183)
0810 22:34:20.165922 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:20.165926 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:20.165938 (+ 12us) server_negotiation.cc:407] Connection header received
0810 22:34:20.166004 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:20.166006 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:20.166060 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:20.166174 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:34:20.166459 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:20.166969 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:20.167940 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:20.168174 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:20.168299 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:20.168625 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:34:20.169150 (+ 525us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:34:20.169289 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:20.169367 (+ 78us) server_negotiation.cc:299] Negotiation successful
0810 22:34:20.169431 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 22:34:20.173764 26123 raft_consensus.cc:1261] T 3ff9dc9e14bb40bbb6d8aad7de5453d3 P 4bcacb667ac14937b041f7d4694b907b [term 1 FOLLOWER]: Refusing update from remote peer c33ed31d8faf4bc89a305efea7fa55a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:34:20.173817 25816 raft_consensus.cc:1261] T 3ff9dc9e14bb40bbb6d8aad7de5453d3 P bd1ece4b27ae4e6c9c715a0045bac33c [term 1 FOLLOWER]: Refusing update from remote peer c33ed31d8faf4bc89a305efea7fa55a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:34:20.174355 26255 consensus_queue.cc:1035] T 3ff9dc9e14bb40bbb6d8aad7de5453d3 P c33ed31d8faf4bc89a305efea7fa55a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bcacb667ac14937b041f7d4694b907b" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 41239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:34:20.174542 26218 consensus_queue.cc:1035] T 3ff9dc9e14bb40bbb6d8aad7de5453d3 P c33ed31d8faf4bc89a305efea7fa55a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:34:20.176290 26213 tablet_replica.cc:439] T 3ff9dc9e14bb40bbb6d8aad7de5453d3 P 4bcacb667ac14937b041f7d4694b907b: TxnStatusTablet state changed. Reason: New leader c33ed31d8faf4bc89a305efea7fa55a9. Latest consensus state: current_term: 1 leader_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33405 } } peers { permanent_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39183 } } peers { permanent_uuid: "4bcacb667ac14937b041f7d4694b907b" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 41239 } } }
I20260810 22:34:20.176432 26213 tablet_replica.cc:442] T 3ff9dc9e14bb40bbb6d8aad7de5453d3 P 4bcacb667ac14937b041f7d4694b907b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 22:34:20.180218 26228 tablet_replica.cc:439] T 3ff9dc9e14bb40bbb6d8aad7de5453d3 P bd1ece4b27ae4e6c9c715a0045bac33c: TxnStatusTablet state changed. Reason: New leader c33ed31d8faf4bc89a305efea7fa55a9. Latest consensus state: current_term: 1 leader_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33405 } } peers { permanent_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39183 } } peers { permanent_uuid: "4bcacb667ac14937b041f7d4694b907b" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 41239 } } }
I20260810 22:34:20.180377 26228 tablet_replica.cc:442] T 3ff9dc9e14bb40bbb6d8aad7de5453d3 P bd1ece4b27ae4e6c9c715a0045bac33c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 22:34:20.180510 26218 tablet_replica.cc:439] T 3ff9dc9e14bb40bbb6d8aad7de5453d3 P c33ed31d8faf4bc89a305efea7fa55a9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33405 } } peers { permanent_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39183 } } peers { permanent_uuid: "4bcacb667ac14937b041f7d4694b907b" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 41239 } } }
I20260810 22:34:20.180704 26218 tablet_replica.cc:442] T 3ff9dc9e14bb40bbb6d8aad7de5453d3 P c33ed31d8faf4bc89a305efea7fa55a9: This TxnStatusTablet replica's current role is: LEADER
I20260810 22:34:20.181350 26213 tablet_replica.cc:439] T 3ff9dc9e14bb40bbb6d8aad7de5453d3 P 4bcacb667ac14937b041f7d4694b907b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33405 } } peers { permanent_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39183 } } peers { permanent_uuid: "4bcacb667ac14937b041f7d4694b907b" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 41239 } } }
I20260810 22:34:20.181468 26213 tablet_replica.cc:442] T 3ff9dc9e14bb40bbb6d8aad7de5453d3 P 4bcacb667ac14937b041f7d4694b907b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 22:34:20.184845 26218 tablet_replica.cc:439] T 3ff9dc9e14bb40bbb6d8aad7de5453d3 P c33ed31d8faf4bc89a305efea7fa55a9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33405 } } peers { permanent_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39183 } } peers { permanent_uuid: "4bcacb667ac14937b041f7d4694b907b" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 41239 } } }
I20260810 22:34:20.184978 26218 tablet_replica.cc:442] T 3ff9dc9e14bb40bbb6d8aad7de5453d3 P c33ed31d8faf4bc89a305efea7fa55a9: This TxnStatusTablet replica's current role is: LEADER
I20260810 22:34:20.185822 26221 mvcc.cc:205] Tried to move back new op lower bound from 7317099561665556480 to 7317099561573949440. Current Snapshot: MvccSnapshot[applied={T|T < 7317099561665556480}]
I20260810 22:34:20.186056 26228 tablet_replica.cc:439] T 3ff9dc9e14bb40bbb6d8aad7de5453d3 P bd1ece4b27ae4e6c9c715a0045bac33c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33405 } } peers { permanent_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39183 } } peers { permanent_uuid: "4bcacb667ac14937b041f7d4694b907b" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 41239 } } }
I20260810 22:34:20.186189 26228 tablet_replica.cc:442] T 3ff9dc9e14bb40bbb6d8aad7de5453d3 P bd1ece4b27ae4e6c9c715a0045bac33c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 22:34:20.192293 26203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:20.186796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56904 (local address 127.16.233.67:41239)
0810 22:34:20.187012 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:20.187016 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:20.187028 (+ 12us) server_negotiation.cc:407] Connection header received
0810 22:34:20.187098 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:20.187100 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:20.187154 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:20.187291 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:34:20.187584 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:20.188351 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:20.190451 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:20.190803 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:20.190946 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:20.191265 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:34:20.191800 (+ 535us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:34:20.191950 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:20.192037 (+ 87us) server_negotiation.cc:299] Negotiation successful
0810 22:34:20.192104 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143}
I20260810 22:34:20.204738 26202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:20.201083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:59289 (local address 127.16.233.66:39183)
0810 22:34:20.201225 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:20.201230 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:20.201317 (+ 87us) server_negotiation.cc:407] Connection header received
0810 22:34:20.201482 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:20.201485 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:20.201535 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:20.201646 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:34:20.201951 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:20.202528 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:20.203506 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:20.203753 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:20.203870 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:20.204235 (+ 365us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:34:20.204318 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:34:20.204416 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:20.204516 (+ 100us) server_negotiation.cc:299] Negotiation successful
0810 22:34:20.204575 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 22:34:20.204738 26183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:20.201030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.66:39183 (local address 127.16.233.67:59289)
0810 22:34:20.201206 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:20.201234 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:20.201349 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:20.201659 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:20.201663 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:20.201671 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 22:34:20.201822 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:20.201828 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:20.202657 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:20.202661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:20.203351 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:20.203361 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:20.203889 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:20.203903 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:20.204017 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:20.204071 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 22:34:20.204421 (+ 350us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 22:34:20.204431 (+ 10us) client_negotiation.cc:769] Sending connection context
0810 22:34:20.204504 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 22:34:20.204597 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260810 22:34:20.213382 25816 raft_consensus.cc:1261] T 69b2cef8db6148a5b2317c81fe95f0a1 P bd1ece4b27ae4e6c9c715a0045bac33c [term 1 FOLLOWER]: Refusing update from remote peer 4bcacb667ac14937b041f7d4694b907b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:34:20.213848 25971 raft_consensus.cc:1261] T 69b2cef8db6148a5b2317c81fe95f0a1 P c33ed31d8faf4bc89a305efea7fa55a9 [term 1 FOLLOWER]: Refusing update from remote peer 4bcacb667ac14937b041f7d4694b907b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:34:20.213960 26239 consensus_queue.cc:1035] T 69b2cef8db6148a5b2317c81fe95f0a1 P 4bcacb667ac14937b041f7d4694b907b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:34:20.214377 26213 consensus_queue.cc:1035] T 69b2cef8db6148a5b2317c81fe95f0a1 P 4bcacb667ac14937b041f7d4694b907b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:34:20.245623 26203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:20.240576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:46415 (local address 127.16.233.67:41239)
0810 22:34:20.240725 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:20.240730 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:20.241036 (+ 306us) server_negotiation.cc:407] Connection header received
0810 22:34:20.241285 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:20.241289 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:20.241363 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:20.241494 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:34:20.241826 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:20.242707 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:20.244050 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:20.244442 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:20.244564 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:20.244995 (+ 431us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:34:20.245096 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:34:20.245182 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:20.245410 (+ 228us) server_negotiation.cc:299] Negotiation successful
0810 22:34:20.245472 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 22:34:20.245733 26268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:20.240428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.67:41239 (local address 127.16.233.66:46415)
0810 22:34:20.240942 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:20.240959 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:20.241137 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:20.241522 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:20.241525 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:20.241534 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 22:34:20.241685 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:20.241692 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:20.242875 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:20.242878 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:20.243897 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:20.243909 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:20.244601 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:20.244604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:20.244683 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:20.244743 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 22:34:20.245241 (+ 498us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 22:34:20.245249 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 22:34:20.245399 (+ 150us) client_negotiation.cc:240] Negotiation successful
0810 22:34:20.245497 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":122,"threads_started":1}
I20260810 22:34:20.277819 25638 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45452:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:34:20.278048 25638 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45452:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:34:20.280848 25638 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 98283e123558435d982c4c04a9e1d6a0: Sending DeleteTablet for 3 replicas of tablet 69b2cef8db6148a5b2317c81fe95f0a1
I20260810 22:34:20.281709 26101 tablet_service.cc:1514] Processing DeleteTablet for tablet 69b2cef8db6148a5b2317c81fe95f0a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:34:20 UTC) from {username='oryx', principal='oryx/127.16.233.126@KRBTEST.COM'} at 127.0.0.1:56890
I20260810 22:34:20.281960 26234 tablet_replica.cc:330] T 69b2cef8db6148a5b2317c81fe95f0a1 P 4bcacb667ac14937b041f7d4694b907b: stopping tablet replica
I20260810 22:34:20.282123 26234 raft_consensus.cc:2229] T 69b2cef8db6148a5b2317c81fe95f0a1 P 4bcacb667ac14937b041f7d4694b907b [term 1 LEADER]: Raft consensus shutting down.
I20260810 22:34:20.282145 25941 tablet_service.cc:1514] Processing DeleteTablet for tablet 69b2cef8db6148a5b2317c81fe95f0a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:34:20 UTC) from {username='oryx', principal='oryx/127.16.233.126@KRBTEST.COM'} at 127.0.0.1:34228
I20260810 22:34:20.282167 25793 tablet_service.cc:1514] Processing DeleteTablet for tablet 69b2cef8db6148a5b2317c81fe95f0a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:34:20 UTC) from {username='oryx', principal='oryx/127.16.233.126@KRBTEST.COM'} at 127.0.0.1:57154
I20260810 22:34:20.282341 26232 tablet_replica.cc:330] T 69b2cef8db6148a5b2317c81fe95f0a1 P c33ed31d8faf4bc89a305efea7fa55a9: stopping tablet replica
I20260810 22:34:20.282451 26232 raft_consensus.cc:2229] T 69b2cef8db6148a5b2317c81fe95f0a1 P c33ed31d8faf4bc89a305efea7fa55a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:34:20.282517 26234 raft_consensus.cc:2258] T 69b2cef8db6148a5b2317c81fe95f0a1 P 4bcacb667ac14937b041f7d4694b907b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:34:20.282588 26232 raft_consensus.cc:2258] T 69b2cef8db6148a5b2317c81fe95f0a1 P c33ed31d8faf4bc89a305efea7fa55a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:34:20.282748 26233 tablet_replica.cc:330] T 69b2cef8db6148a5b2317c81fe95f0a1 P bd1ece4b27ae4e6c9c715a0045bac33c: stopping tablet replica
I20260810 22:34:20.282908 26233 raft_consensus.cc:2229] T 69b2cef8db6148a5b2317c81fe95f0a1 P bd1ece4b27ae4e6c9c715a0045bac33c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:34:20.283051 26233 raft_consensus.cc:2258] T 69b2cef8db6148a5b2317c81fe95f0a1 P bd1ece4b27ae4e6c9c715a0045bac33c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:34:20.283919 26232 ts_tablet_manager.cc:1905] T 69b2cef8db6148a5b2317c81fe95f0a1 P c33ed31d8faf4bc89a305efea7fa55a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:34:20.284034 26233 ts_tablet_manager.cc:1905] T 69b2cef8db6148a5b2317c81fe95f0a1 P bd1ece4b27ae4e6c9c715a0045bac33c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:34:20.285631 26234 ts_tablet_manager.cc:1905] T 69b2cef8db6148a5b2317c81fe95f0a1 P 4bcacb667ac14937b041f7d4694b907b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:34:20.287432 26232 ts_tablet_manager.cc:1918] T 69b2cef8db6148a5b2317c81fe95f0a1 P c33ed31d8faf4bc89a305efea7fa55a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 22:34:20.287542 26232 log.cc:1196] T 69b2cef8db6148a5b2317c81fe95f0a1 P c33ed31d8faf4bc89a305efea7fa55a9: Deleting WAL directory at /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/ts-1/wal/wals/69b2cef8db6148a5b2317c81fe95f0a1
I20260810 22:34:20.287837 26232 ts_tablet_manager.cc:1939] T 69b2cef8db6148a5b2317c81fe95f0a1 P c33ed31d8faf4bc89a305efea7fa55a9: Deleting consensus metadata
I20260810 22:34:20.289340 26234 ts_tablet_manager.cc:1918] T 69b2cef8db6148a5b2317c81fe95f0a1 P 4bcacb667ac14937b041f7d4694b907b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 22:34:20.289459 26234 log.cc:1196] T 69b2cef8db6148a5b2317c81fe95f0a1 P 4bcacb667ac14937b041f7d4694b907b: Deleting WAL directory at /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/ts-2/wal/wals/69b2cef8db6148a5b2317c81fe95f0a1
I20260810 22:34:20.289438 25620 catalog_manager.cc:4841] TS c33ed31d8faf4bc89a305efea7fa55a9 (127.16.233.66:39183): tablet 69b2cef8db6148a5b2317c81fe95f0a1 (table test-table [id=709c494f628f493baa19db2d61e3152b]) successfully deleted
I20260810 22:34:20.289767 26234 ts_tablet_manager.cc:1939] T 69b2cef8db6148a5b2317c81fe95f0a1 P 4bcacb667ac14937b041f7d4694b907b: Deleting consensus metadata
I20260810 22:34:20.293001 26233 ts_tablet_manager.cc:1918] T 69b2cef8db6148a5b2317c81fe95f0a1 P bd1ece4b27ae4e6c9c715a0045bac33c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 22:34:20.293126 26233 log.cc:1196] T 69b2cef8db6148a5b2317c81fe95f0a1 P bd1ece4b27ae4e6c9c715a0045bac33c: Deleting WAL directory at /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/ts-0/wal/wals/69b2cef8db6148a5b2317c81fe95f0a1
I20260810 22:34:20.293452 26233 ts_tablet_manager.cc:1939] T 69b2cef8db6148a5b2317c81fe95f0a1 P bd1ece4b27ae4e6c9c715a0045bac33c: Deleting consensus metadata
I20260810 22:34:20.294498 25620 catalog_manager.cc:4841] TS bd1ece4b27ae4e6c9c715a0045bac33c (127.16.233.65:33405): tablet 69b2cef8db6148a5b2317c81fe95f0a1 (table test-table [id=709c494f628f493baa19db2d61e3152b]) successfully deleted
I20260810 22:34:20.295017 25621 catalog_manager.cc:4841] TS 4bcacb667ac14937b041f7d4694b907b (127.16.233.67:41239): tablet 69b2cef8db6148a5b2317c81fe95f0a1 (table test-table [id=709c494f628f493baa19db2d61e3152b]) successfully deleted
Aug 10 22:34:20 dist-test-slave-77v9 krb5kdc[25588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 22:34:20 dist-test-slave-77v9 krb5kdc[25588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:34:20 dist-test-slave-77v9 krb5kdc[25588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
W20260810 22:34:20.331218 25695 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:34:20.321636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45468 (local address 127.16.233.126:45483)
0810 22:34:20.321777 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:20.321782 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:20.322130 (+ 348us) server_negotiation.cc:407] Connection header received
0810 22:34:20.322261 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:20.322266 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:20.322337 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:20.322447 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:20.323468 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:20.324410 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:20.325835 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:20.326230 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:20.326398 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:20.331057 (+ 4659us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45468: BlockingRecv error: recv got EOF from 127.0.0.1:45468 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 10 22:34:20 dist-test-slave-77v9 krb5kdc[25588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM
I20260810 22:34:20.345471 25695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:20.332889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45480 (local address 127.16.233.126:45483)
0810 22:34:20.333023 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:20.333028 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:20.333556 (+ 528us) server_negotiation.cc:407] Connection header received
0810 22:34:20.333710 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:20.333713 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:20.333769 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:20.333859 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:20.334825 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:20.335688 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:20.337101 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:20.337422 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:20.337540 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:20.341080 (+ 3540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:20.341085 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:20.341088 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:20.341134 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:20.343110 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:20.343776 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:20.343780 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:20.343782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:20.343905 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:20.344282 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:20.344286 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:20.344288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:20.344565 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:20.344713 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:20.345114 (+ 401us) server_negotiation.cc:299] Negotiation successful
0810 22:34:20.345279 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 22:34:20.349377 25638 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45480:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 22:34:20.349960 25638 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:34:20.356470 25793 tablet_service.cc:1467] Processing CreateTablet for tablet 2cd544dcbb5e407d9bf6ada0053028a6 (DEFAULT_TABLE table=test-table [id=26fe384eec944b85815330e16471fc43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:34:20.356576 25941 tablet_service.cc:1467] Processing CreateTablet for tablet 2cd544dcbb5e407d9bf6ada0053028a6 (DEFAULT_TABLE table=test-table [id=26fe384eec944b85815330e16471fc43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:34:20.356957 25941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cd544dcbb5e407d9bf6ada0053028a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:34:20.357182 25793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cd544dcbb5e407d9bf6ada0053028a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:34:20.360220 26206 tablet_bootstrap.cc:492] T 2cd544dcbb5e407d9bf6ada0053028a6 P bd1ece4b27ae4e6c9c715a0045bac33c: Bootstrap starting.
I20260810 22:34:20.360203 26101 tablet_service.cc:1467] Processing CreateTablet for tablet 2cd544dcbb5e407d9bf6ada0053028a6 (DEFAULT_TABLE table=test-table [id=26fe384eec944b85815330e16471fc43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:34:20.360524 26101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cd544dcbb5e407d9bf6ada0053028a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:34:20.361228 26206 tablet_bootstrap.cc:654] T 2cd544dcbb5e407d9bf6ada0053028a6 P bd1ece4b27ae4e6c9c715a0045bac33c: Neither blocks nor log segments found. Creating new log.
I20260810 22:34:20.363399 26208 tablet_bootstrap.cc:492] T 2cd544dcbb5e407d9bf6ada0053028a6 P 4bcacb667ac14937b041f7d4694b907b: Bootstrap starting.
I20260810 22:34:20.364451 26208 tablet_bootstrap.cc:654] T 2cd544dcbb5e407d9bf6ada0053028a6 P 4bcacb667ac14937b041f7d4694b907b: Neither blocks nor log segments found. Creating new log.
I20260810 22:34:20.366266 26208 tablet_bootstrap.cc:492] T 2cd544dcbb5e407d9bf6ada0053028a6 P 4bcacb667ac14937b041f7d4694b907b: No bootstrap required, opened a new log
I20260810 22:34:20.366367 26208 ts_tablet_manager.cc:1397] T 2cd544dcbb5e407d9bf6ada0053028a6 P 4bcacb667ac14937b041f7d4694b907b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 22:34:20.366906 26208 raft_consensus.cc:348] T 2cd544dcbb5e407d9bf6ada0053028a6 P 4bcacb667ac14937b041f7d4694b907b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33405 } } peers { permanent_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39183 } } peers { permanent_uuid: "4bcacb667ac14937b041f7d4694b907b" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 41239 } }
I20260810 22:34:20.367049 26208 raft_consensus.cc:374] T 2cd544dcbb5e407d9bf6ada0053028a6 P 4bcacb667ac14937b041f7d4694b907b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:34:20.367082 26208 raft_consensus.cc:729] T 2cd544dcbb5e407d9bf6ada0053028a6 P 4bcacb667ac14937b041f7d4694b907b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bcacb667ac14937b041f7d4694b907b, State: Initialized, Role: FOLLOWER
I20260810 22:34:20.367216 26208 consensus_queue.cc:260] T 2cd544dcbb5e407d9bf6ada0053028a6 P 4bcacb667ac14937b041f7d4694b907b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33405 } } peers { permanent_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39183 } } peers { permanent_uuid: "4bcacb667ac14937b041f7d4694b907b" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 41239 } }
I20260810 22:34:20.367507 26208 ts_tablet_manager.cc:1428] T 2cd544dcbb5e407d9bf6ada0053028a6 P 4bcacb667ac14937b041f7d4694b907b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 22:34:20.367748 26207 tablet_bootstrap.cc:492] T 2cd544dcbb5e407d9bf6ada0053028a6 P c33ed31d8faf4bc89a305efea7fa55a9: Bootstrap starting.
I20260810 22:34:20.368727 26206 tablet_bootstrap.cc:492] T 2cd544dcbb5e407d9bf6ada0053028a6 P bd1ece4b27ae4e6c9c715a0045bac33c: No bootstrap required, opened a new log
I20260810 22:34:20.368988 26207 tablet_bootstrap.cc:654] T 2cd544dcbb5e407d9bf6ada0053028a6 P c33ed31d8faf4bc89a305efea7fa55a9: Neither blocks nor log segments found. Creating new log.
I20260810 22:34:20.369055 26206 ts_tablet_manager.cc:1397] T 2cd544dcbb5e407d9bf6ada0053028a6 P bd1ece4b27ae4e6c9c715a0045bac33c: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260810 22:34:20.369532 26206 raft_consensus.cc:348] T 2cd544dcbb5e407d9bf6ada0053028a6 P bd1ece4b27ae4e6c9c715a0045bac33c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33405 } } peers { permanent_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39183 } } peers { permanent_uuid: "4bcacb667ac14937b041f7d4694b907b" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 41239 } }
I20260810 22:34:20.369668 26206 raft_consensus.cc:374] T 2cd544dcbb5e407d9bf6ada0053028a6 P bd1ece4b27ae4e6c9c715a0045bac33c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:34:20.369695 26206 raft_consensus.cc:729] T 2cd544dcbb5e407d9bf6ada0053028a6 P bd1ece4b27ae4e6c9c715a0045bac33c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd1ece4b27ae4e6c9c715a0045bac33c, State: Initialized, Role: FOLLOWER
I20260810 22:34:20.369833 26206 consensus_queue.cc:260] T 2cd544dcbb5e407d9bf6ada0053028a6 P bd1ece4b27ae4e6c9c715a0045bac33c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33405 } } peers { permanent_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39183 } } peers { permanent_uuid: "4bcacb667ac14937b041f7d4694b907b" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 41239 } }
I20260810 22:34:20.370081 26206 ts_tablet_manager.cc:1428] T 2cd544dcbb5e407d9bf6ada0053028a6 P bd1ece4b27ae4e6c9c715a0045bac33c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 22:34:20.371380 26207 tablet_bootstrap.cc:492] T 2cd544dcbb5e407d9bf6ada0053028a6 P c33ed31d8faf4bc89a305efea7fa55a9: No bootstrap required, opened a new log
I20260810 22:34:20.371479 26207 ts_tablet_manager.cc:1397] T 2cd544dcbb5e407d9bf6ada0053028a6 P c33ed31d8faf4bc89a305efea7fa55a9: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260810 22:34:20.371929 26207 raft_consensus.cc:348] T 2cd544dcbb5e407d9bf6ada0053028a6 P c33ed31d8faf4bc89a305efea7fa55a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33405 } } peers { permanent_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39183 } } peers { permanent_uuid: "4bcacb667ac14937b041f7d4694b907b" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 41239 } }
I20260810 22:34:20.372056 26207 raft_consensus.cc:374] T 2cd544dcbb5e407d9bf6ada0053028a6 P c33ed31d8faf4bc89a305efea7fa55a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:34:20.372087 26207 raft_consensus.cc:729] T 2cd544dcbb5e407d9bf6ada0053028a6 P c33ed31d8faf4bc89a305efea7fa55a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c33ed31d8faf4bc89a305efea7fa55a9, State: Initialized, Role: FOLLOWER
I20260810 22:34:20.372231 26207 consensus_queue.cc:260] T 2cd544dcbb5e407d9bf6ada0053028a6 P c33ed31d8faf4bc89a305efea7fa55a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33405 } } peers { permanent_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39183 } } peers { permanent_uuid: "4bcacb667ac14937b041f7d4694b907b" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 41239 } }
I20260810 22:34:20.372493 26207 ts_tablet_manager.cc:1428] T 2cd544dcbb5e407d9bf6ada0053028a6 P c33ed31d8faf4bc89a305efea7fa55a9: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 22:34:20.552691 26228 raft_consensus.cc:482] T 2cd544dcbb5e407d9bf6ada0053028a6 P bd1ece4b27ae4e6c9c715a0045bac33c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:34:20.552963 26228 raft_consensus.cc:504] T 2cd544dcbb5e407d9bf6ada0053028a6 P bd1ece4b27ae4e6c9c715a0045bac33c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33405 } } peers { permanent_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39183 } } peers { permanent_uuid: "4bcacb667ac14937b041f7d4694b907b" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 41239 } }
I20260810 22:34:20.553560 26228 leader_election.cc:288] T 2cd544dcbb5e407d9bf6ada0053028a6 P bd1ece4b27ae4e6c9c715a0045bac33c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c33ed31d8faf4bc89a305efea7fa55a9 (127.16.233.66:39183), 4bcacb667ac14937b041f7d4694b907b (127.16.233.67:41239)
I20260810 22:34:20.554123 25971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd544dcbb5e407d9bf6ada0053028a6" candidate_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" is_pre_election: true
I20260810 22:34:20.554251 26123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd544dcbb5e407d9bf6ada0053028a6" candidate_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bcacb667ac14937b041f7d4694b907b" is_pre_election: true
I20260810 22:34:20.554322 25971 raft_consensus.cc:2454] T 2cd544dcbb5e407d9bf6ada0053028a6 P c33ed31d8faf4bc89a305efea7fa55a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd1ece4b27ae4e6c9c715a0045bac33c in term 0.
I20260810 22:34:20.554410 26123 raft_consensus.cc:2454] T 2cd544dcbb5e407d9bf6ada0053028a6 P 4bcacb667ac14937b041f7d4694b907b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd1ece4b27ae4e6c9c715a0045bac33c in term 0.
I20260810 22:34:20.554724 25739 leader_election.cc:302] T 2cd544dcbb5e407d9bf6ada0053028a6 P bd1ece4b27ae4e6c9c715a0045bac33c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd1ece4b27ae4e6c9c715a0045bac33c, c33ed31d8faf4bc89a305efea7fa55a9; no voters:
I20260810 22:34:20.554972 26228 raft_consensus.cc:2781] T 2cd544dcbb5e407d9bf6ada0053028a6 P bd1ece4b27ae4e6c9c715a0045bac33c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:34:20.555042 26228 raft_consensus.cc:482] T 2cd544dcbb5e407d9bf6ada0053028a6 P bd1ece4b27ae4e6c9c715a0045bac33c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:34:20.555085 26228 raft_consensus.cc:3037] T 2cd544dcbb5e407d9bf6ada0053028a6 P bd1ece4b27ae4e6c9c715a0045bac33c [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:34:20.556224 26228 raft_consensus.cc:504] T 2cd544dcbb5e407d9bf6ada0053028a6 P bd1ece4b27ae4e6c9c715a0045bac33c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33405 } } peers { permanent_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39183 } } peers { permanent_uuid: "4bcacb667ac14937b041f7d4694b907b" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 41239 } }
I20260810 22:34:20.556617 26228 leader_election.cc:288] T 2cd544dcbb5e407d9bf6ada0053028a6 P bd1ece4b27ae4e6c9c715a0045bac33c [CANDIDATE]: Term 1 election: Requested vote from peers c33ed31d8faf4bc89a305efea7fa55a9 (127.16.233.66:39183), 4bcacb667ac14937b041f7d4694b907b (127.16.233.67:41239)
I20260810 22:34:20.557126 26123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd544dcbb5e407d9bf6ada0053028a6" candidate_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bcacb667ac14937b041f7d4694b907b"
I20260810 22:34:20.557104 25971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cd544dcbb5e407d9bf6ada0053028a6" candidate_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c33ed31d8faf4bc89a305efea7fa55a9"
I20260810 22:34:20.557283 26123 raft_consensus.cc:3037] T 2cd544dcbb5e407d9bf6ada0053028a6 P 4bcacb667ac14937b041f7d4694b907b [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:34:20.557298 25971 raft_consensus.cc:3037] T 2cd544dcbb5e407d9bf6ada0053028a6 P c33ed31d8faf4bc89a305efea7fa55a9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:34:20.558303 25971 raft_consensus.cc:2454] T 2cd544dcbb5e407d9bf6ada0053028a6 P c33ed31d8faf4bc89a305efea7fa55a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd1ece4b27ae4e6c9c715a0045bac33c in term 1.
I20260810 22:34:20.558310 26123 raft_consensus.cc:2454] T 2cd544dcbb5e407d9bf6ada0053028a6 P 4bcacb667ac14937b041f7d4694b907b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd1ece4b27ae4e6c9c715a0045bac33c in term 1.
I20260810 22:34:20.558668 25739 leader_election.cc:302] T 2cd544dcbb5e407d9bf6ada0053028a6 P bd1ece4b27ae4e6c9c715a0045bac33c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4bcacb667ac14937b041f7d4694b907b, bd1ece4b27ae4e6c9c715a0045bac33c, c33ed31d8faf4bc89a305efea7fa55a9; no voters:
I20260810 22:34:20.558851 26228 raft_consensus.cc:2781] T 2cd544dcbb5e407d9bf6ada0053028a6 P bd1ece4b27ae4e6c9c715a0045bac33c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:34:20.559053 26228 raft_consensus.cc:686] T 2cd544dcbb5e407d9bf6ada0053028a6 P bd1ece4b27ae4e6c9c715a0045bac33c [term 1 LEADER]: Becoming Leader. State: Replica: bd1ece4b27ae4e6c9c715a0045bac33c, State: Running, Role: LEADER
I20260810 22:34:20.559258 26228 consensus_queue.cc:237] T 2cd544dcbb5e407d9bf6ada0053028a6 P bd1ece4b27ae4e6c9c715a0045bac33c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33405 } } peers { permanent_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39183 } } peers { permanent_uuid: "4bcacb667ac14937b041f7d4694b907b" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 41239 } }
I20260810 22:34:20.561215 25637 catalog_manager.cc:5495] T 2cd544dcbb5e407d9bf6ada0053028a6 P bd1ece4b27ae4e6c9c715a0045bac33c reported cstate change: term changed from 0 to 1, leader changed from <none> to bd1ece4b27ae4e6c9c715a0045bac33c (127.16.233.65). New cstate: current_term: 1 leader_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33405 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bcacb667ac14937b041f7d4694b907b" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 41239 } health_report { overall_health: UNKNOWN } } }
I20260810 22:34:20.574862 26205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:20.570690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57164 (local address 127.16.233.65:33405)
0810 22:34:20.570866 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:20.570871 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:20.570946 (+ 75us) server_negotiation.cc:407] Connection header received
0810 22:34:20.571147 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:20.571150 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:20.571206 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:20.571293 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:34:20.571680 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:20.572323 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:20.573580 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:20.573824 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:20.573931 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:20.574253 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:34:20.574333 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:34:20.574411 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:20.574556 (+ 145us) server_negotiation.cc:299] Negotiation successful
0810 22:34:20.574668 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260810 22:34:20.577332 26123 raft_consensus.cc:1261] T 2cd544dcbb5e407d9bf6ada0053028a6 P 4bcacb667ac14937b041f7d4694b907b [term 1 FOLLOWER]: Refusing update from remote peer bd1ece4b27ae4e6c9c715a0045bac33c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:34:20.577451 25971 raft_consensus.cc:1261] T 2cd544dcbb5e407d9bf6ada0053028a6 P c33ed31d8faf4bc89a305efea7fa55a9 [term 1 FOLLOWER]: Refusing update from remote peer bd1ece4b27ae4e6c9c715a0045bac33c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:34:20.577960 26289 consensus_queue.cc:1035] T 2cd544dcbb5e407d9bf6ada0053028a6 P bd1ece4b27ae4e6c9c715a0045bac33c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bcacb667ac14937b041f7d4694b907b" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 41239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:34:20.578173 26228 consensus_queue.cc:1035] T 2cd544dcbb5e407d9bf6ada0053028a6 P bd1ece4b27ae4e6c9c715a0045bac33c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:34:20.587044 25637 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:34:20.587248 25637 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:34:20.589731 25637 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 98283e123558435d982c4c04a9e1d6a0: Sending DeleteTablet for 3 replicas of tablet 2cd544dcbb5e407d9bf6ada0053028a6
I20260810 22:34:20.590430 25793 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cd544dcbb5e407d9bf6ada0053028a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:34:20 UTC) from {username='oryx', principal='oryx/127.16.233.126@KRBTEST.COM'} at 127.0.0.1:57154
I20260810 22:34:20.590577 25941 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cd544dcbb5e407d9bf6ada0053028a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:34:20 UTC) from {username='oryx', principal='oryx/127.16.233.126@KRBTEST.COM'} at 127.0.0.1:34228
I20260810 22:34:20.590631 26101 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cd544dcbb5e407d9bf6ada0053028a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:34:20 UTC) from {username='oryx', principal='oryx/127.16.233.126@KRBTEST.COM'} at 127.0.0.1:56890
I20260810 22:34:20.590813 26233 tablet_replica.cc:330] T 2cd544dcbb5e407d9bf6ada0053028a6 P bd1ece4b27ae4e6c9c715a0045bac33c: stopping tablet replica
I20260810 22:34:20.590852 26234 tablet_replica.cc:330] T 2cd544dcbb5e407d9bf6ada0053028a6 P 4bcacb667ac14937b041f7d4694b907b: stopping tablet replica
I20260810 22:34:20.590845 26232 tablet_replica.cc:330] T 2cd544dcbb5e407d9bf6ada0053028a6 P c33ed31d8faf4bc89a305efea7fa55a9: stopping tablet replica
I20260810 22:34:20.590968 26234 raft_consensus.cc:2229] T 2cd544dcbb5e407d9bf6ada0053028a6 P 4bcacb667ac14937b041f7d4694b907b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:34:20.590970 26233 raft_consensus.cc:2229] T 2cd544dcbb5e407d9bf6ada0053028a6 P bd1ece4b27ae4e6c9c715a0045bac33c [term 1 LEADER]: Raft consensus shutting down.
I20260810 22:34:20.590974 26232 raft_consensus.cc:2229] T 2cd544dcbb5e407d9bf6ada0053028a6 P c33ed31d8faf4bc89a305efea7fa55a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:34:20.591117 26234 raft_consensus.cc:2258] T 2cd544dcbb5e407d9bf6ada0053028a6 P 4bcacb667ac14937b041f7d4694b907b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:34:20.591125 26232 raft_consensus.cc:2258] T 2cd544dcbb5e407d9bf6ada0053028a6 P c33ed31d8faf4bc89a305efea7fa55a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:34:20.591192 26233 raft_consensus.cc:2258] T 2cd544dcbb5e407d9bf6ada0053028a6 P bd1ece4b27ae4e6c9c715a0045bac33c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:34:20.591600 25637 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45480:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 22:34:20.591997 26232 ts_tablet_manager.cc:1905] T 2cd544dcbb5e407d9bf6ada0053028a6 P c33ed31d8faf4bc89a305efea7fa55a9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 22:34:20.592131 25637 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:34:20.592236 26234 ts_tablet_manager.cc:1905] T 2cd544dcbb5e407d9bf6ada0053028a6 P 4bcacb667ac14937b041f7d4694b907b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:34:20.595057 26232 ts_tablet_manager.cc:1918] T 2cd544dcbb5e407d9bf6ada0053028a6 P c33ed31d8faf4bc89a305efea7fa55a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:34:20.595211 26232 log.cc:1196] T 2cd544dcbb5e407d9bf6ada0053028a6 P c33ed31d8faf4bc89a305efea7fa55a9: Deleting WAL directory at /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/ts-1/wal/wals/2cd544dcbb5e407d9bf6ada0053028a6
I20260810 22:34:20.595299 26234 ts_tablet_manager.cc:1918] T 2cd544dcbb5e407d9bf6ada0053028a6 P 4bcacb667ac14937b041f7d4694b907b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:34:20.595396 26234 log.cc:1196] T 2cd544dcbb5e407d9bf6ada0053028a6 P 4bcacb667ac14937b041f7d4694b907b: Deleting WAL directory at /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/ts-2/wal/wals/2cd544dcbb5e407d9bf6ada0053028a6
I20260810 22:34:20.595521 26232 ts_tablet_manager.cc:1939] T 2cd544dcbb5e407d9bf6ada0053028a6 P c33ed31d8faf4bc89a305efea7fa55a9: Deleting consensus metadata
I20260810 22:34:20.595718 26234 ts_tablet_manager.cc:1939] T 2cd544dcbb5e407d9bf6ada0053028a6 P 4bcacb667ac14937b041f7d4694b907b: Deleting consensus metadata
I20260810 22:34:20.596370 25620 catalog_manager.cc:4841] TS c33ed31d8faf4bc89a305efea7fa55a9 (127.16.233.66:39183): tablet 2cd544dcbb5e407d9bf6ada0053028a6 (table test-table [id=26fe384eec944b85815330e16471fc43]) successfully deleted
I20260810 22:34:20.596697 25621 catalog_manager.cc:4841] TS 4bcacb667ac14937b041f7d4694b907b (127.16.233.67:41239): tablet 2cd544dcbb5e407d9bf6ada0053028a6 (table test-table [id=26fe384eec944b85815330e16471fc43]) successfully deleted
I20260810 22:34:20.597039 26233 ts_tablet_manager.cc:1905] T 2cd544dcbb5e407d9bf6ada0053028a6 P bd1ece4b27ae4e6c9c715a0045bac33c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:34:20.599027 26101 tablet_service.cc:1467] Processing CreateTablet for tablet 5a3330e783344a6c86246dfba1399aab (DEFAULT_TABLE table=test-table [id=1270c0983d7f46c18fb6869898881700]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:34:20.599175 25793 tablet_service.cc:1467] Processing CreateTablet for tablet 5a3330e783344a6c86246dfba1399aab (DEFAULT_TABLE table=test-table [id=1270c0983d7f46c18fb6869898881700]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:34:20.599387 26101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a3330e783344a6c86246dfba1399aab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:34:20.599483 25793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a3330e783344a6c86246dfba1399aab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:34:20.600184 26233 ts_tablet_manager.cc:1918] T 2cd544dcbb5e407d9bf6ada0053028a6 P bd1ece4b27ae4e6c9c715a0045bac33c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:34:20.600275 26233 log.cc:1196] T 2cd544dcbb5e407d9bf6ada0053028a6 P bd1ece4b27ae4e6c9c715a0045bac33c: Deleting WAL directory at /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/ts-0/wal/wals/2cd544dcbb5e407d9bf6ada0053028a6
I20260810 22:34:20.600590 26233 ts_tablet_manager.cc:1939] T 2cd544dcbb5e407d9bf6ada0053028a6 P bd1ece4b27ae4e6c9c715a0045bac33c: Deleting consensus metadata
I20260810 22:34:20.601861 26208 tablet_bootstrap.cc:492] T 5a3330e783344a6c86246dfba1399aab P 4bcacb667ac14937b041f7d4694b907b: Bootstrap starting.
I20260810 22:34:20.602418 26206 tablet_bootstrap.cc:492] T 5a3330e783344a6c86246dfba1399aab P bd1ece4b27ae4e6c9c715a0045bac33c: Bootstrap starting.
I20260810 22:34:20.602835 26208 tablet_bootstrap.cc:654] T 5a3330e783344a6c86246dfba1399aab P 4bcacb667ac14937b041f7d4694b907b: Neither blocks nor log segments found. Creating new log.
I20260810 22:34:20.603225 26206 tablet_bootstrap.cc:654] T 5a3330e783344a6c86246dfba1399aab P bd1ece4b27ae4e6c9c715a0045bac33c: Neither blocks nor log segments found. Creating new log.
I20260810 22:34:20.603902 25620 catalog_manager.cc:4841] TS bd1ece4b27ae4e6c9c715a0045bac33c (127.16.233.65:33405): tablet 2cd544dcbb5e407d9bf6ada0053028a6 (table test-table [id=26fe384eec944b85815330e16471fc43]) successfully deleted
I20260810 22:34:20.604063 26208 tablet_bootstrap.cc:492] T 5a3330e783344a6c86246dfba1399aab P 4bcacb667ac14937b041f7d4694b907b: No bootstrap required, opened a new log
I20260810 22:34:20.604164 26208 ts_tablet_manager.cc:1397] T 5a3330e783344a6c86246dfba1399aab P 4bcacb667ac14937b041f7d4694b907b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 22:34:20.604449 25941 tablet_service.cc:1467] Processing CreateTablet for tablet 5a3330e783344a6c86246dfba1399aab (DEFAULT_TABLE table=test-table [id=1270c0983d7f46c18fb6869898881700]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:34:20.604543 26206 tablet_bootstrap.cc:492] T 5a3330e783344a6c86246dfba1399aab P bd1ece4b27ae4e6c9c715a0045bac33c: No bootstrap required, opened a new log
I20260810 22:34:20.604629 26206 ts_tablet_manager.cc:1397] T 5a3330e783344a6c86246dfba1399aab P bd1ece4b27ae4e6c9c715a0045bac33c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 22:34:20.604594 26208 raft_consensus.cc:348] T 5a3330e783344a6c86246dfba1399aab P 4bcacb667ac14937b041f7d4694b907b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33405 } } peers { permanent_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39183 } } peers { permanent_uuid: "4bcacb667ac14937b041f7d4694b907b" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 41239 } }
I20260810 22:34:20.604862 25941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a3330e783344a6c86246dfba1399aab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:34:20.604738 26208 raft_consensus.cc:374] T 5a3330e783344a6c86246dfba1399aab P 4bcacb667ac14937b041f7d4694b907b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:34:20.605137 26208 raft_consensus.cc:729] T 5a3330e783344a6c86246dfba1399aab P 4bcacb667ac14937b041f7d4694b907b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bcacb667ac14937b041f7d4694b907b, State: Initialized, Role: FOLLOWER
I20260810 22:34:20.605119 26206 raft_consensus.cc:348] T 5a3330e783344a6c86246dfba1399aab P bd1ece4b27ae4e6c9c715a0045bac33c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33405 } } peers { permanent_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39183 } } peers { permanent_uuid: "4bcacb667ac14937b041f7d4694b907b" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 41239 } }
I20260810 22:34:20.605214 26206 raft_consensus.cc:374] T 5a3330e783344a6c86246dfba1399aab P bd1ece4b27ae4e6c9c715a0045bac33c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:34:20.605237 26206 raft_consensus.cc:729] T 5a3330e783344a6c86246dfba1399aab P bd1ece4b27ae4e6c9c715a0045bac33c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd1ece4b27ae4e6c9c715a0045bac33c, State: Initialized, Role: FOLLOWER
I20260810 22:34:20.605386 26206 consensus_queue.cc:260] T 5a3330e783344a6c86246dfba1399aab P bd1ece4b27ae4e6c9c715a0045bac33c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33405 } } peers { permanent_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39183 } } peers { permanent_uuid: "4bcacb667ac14937b041f7d4694b907b" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 41239 } }
I20260810 22:34:20.605460 26208 consensus_queue.cc:260] T 5a3330e783344a6c86246dfba1399aab P 4bcacb667ac14937b041f7d4694b907b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33405 } } peers { permanent_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39183 } } peers { permanent_uuid: "4bcacb667ac14937b041f7d4694b907b" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 41239 } }
I20260810 22:34:20.605672 26206 ts_tablet_manager.cc:1428] T 5a3330e783344a6c86246dfba1399aab P bd1ece4b27ae4e6c9c715a0045bac33c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 22:34:20.607498 26208 ts_tablet_manager.cc:1428] T 5a3330e783344a6c86246dfba1399aab P 4bcacb667ac14937b041f7d4694b907b: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20260810 22:34:20.608914 26207 tablet_bootstrap.cc:492] T 5a3330e783344a6c86246dfba1399aab P c33ed31d8faf4bc89a305efea7fa55a9: Bootstrap starting.
I20260810 22:34:20.610082 26207 tablet_bootstrap.cc:654] T 5a3330e783344a6c86246dfba1399aab P c33ed31d8faf4bc89a305efea7fa55a9: Neither blocks nor log segments found. Creating new log.
I20260810 22:34:20.611544 26207 tablet_bootstrap.cc:492] T 5a3330e783344a6c86246dfba1399aab P c33ed31d8faf4bc89a305efea7fa55a9: No bootstrap required, opened a new log
I20260810 22:34:20.611636 26207 ts_tablet_manager.cc:1397] T 5a3330e783344a6c86246dfba1399aab P c33ed31d8faf4bc89a305efea7fa55a9: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260810 22:34:20.612006 26207 raft_consensus.cc:348] T 5a3330e783344a6c86246dfba1399aab P c33ed31d8faf4bc89a305efea7fa55a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33405 } } peers { permanent_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39183 } } peers { permanent_uuid: "4bcacb667ac14937b041f7d4694b907b" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 41239 } }
I20260810 22:34:20.612119 26207 raft_consensus.cc:374] T 5a3330e783344a6c86246dfba1399aab P c33ed31d8faf4bc89a305efea7fa55a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:34:20.612143 26207 raft_consensus.cc:729] T 5a3330e783344a6c86246dfba1399aab P c33ed31d8faf4bc89a305efea7fa55a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c33ed31d8faf4bc89a305efea7fa55a9, State: Initialized, Role: FOLLOWER
I20260810 22:34:20.612368 26207 consensus_queue.cc:260] T 5a3330e783344a6c86246dfba1399aab P c33ed31d8faf4bc89a305efea7fa55a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33405 } } peers { permanent_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39183 } } peers { permanent_uuid: "4bcacb667ac14937b041f7d4694b907b" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 41239 } }
I20260810 22:34:20.612828 26207 ts_tablet_manager.cc:1428] T 5a3330e783344a6c86246dfba1399aab P c33ed31d8faf4bc89a305efea7fa55a9: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260810 22:34:20.619768 26255 raft_consensus.cc:482] T 5a3330e783344a6c86246dfba1399aab P c33ed31d8faf4bc89a305efea7fa55a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:34:20.619966 26255 raft_consensus.cc:504] T 5a3330e783344a6c86246dfba1399aab P c33ed31d8faf4bc89a305efea7fa55a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33405 } } peers { permanent_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39183 } } peers { permanent_uuid: "4bcacb667ac14937b041f7d4694b907b" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 41239 } }
I20260810 22:34:20.620463 26255 leader_election.cc:288] T 5a3330e783344a6c86246dfba1399aab P c33ed31d8faf4bc89a305efea7fa55a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd1ece4b27ae4e6c9c715a0045bac33c (127.16.233.65:33405), 4bcacb667ac14937b041f7d4694b907b (127.16.233.67:41239)
I20260810 22:34:20.621047 25816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a3330e783344a6c86246dfba1399aab" candidate_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" is_pre_election: true
I20260810 22:34:20.621150 26123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a3330e783344a6c86246dfba1399aab" candidate_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bcacb667ac14937b041f7d4694b907b" is_pre_election: true
I20260810 22:34:20.621249 25816 raft_consensus.cc:2454] T 5a3330e783344a6c86246dfba1399aab P bd1ece4b27ae4e6c9c715a0045bac33c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c33ed31d8faf4bc89a305efea7fa55a9 in term 0.
I20260810 22:34:20.621302 26123 raft_consensus.cc:2454] T 5a3330e783344a6c86246dfba1399aab P 4bcacb667ac14937b041f7d4694b907b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c33ed31d8faf4bc89a305efea7fa55a9 in term 0.
I20260810 22:34:20.621644 25894 leader_election.cc:302] T 5a3330e783344a6c86246dfba1399aab P c33ed31d8faf4bc89a305efea7fa55a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd1ece4b27ae4e6c9c715a0045bac33c, c33ed31d8faf4bc89a305efea7fa55a9; no voters:
I20260810 22:34:20.622049 26255 raft_consensus.cc:2781] T 5a3330e783344a6c86246dfba1399aab P c33ed31d8faf4bc89a305efea7fa55a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:34:20.622131 26255 raft_consensus.cc:482] T 5a3330e783344a6c86246dfba1399aab P c33ed31d8faf4bc89a305efea7fa55a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:34:20.622185 26255 raft_consensus.cc:3037] T 5a3330e783344a6c86246dfba1399aab P c33ed31d8faf4bc89a305efea7fa55a9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:34:20.623461 26255 raft_consensus.cc:504] T 5a3330e783344a6c86246dfba1399aab P c33ed31d8faf4bc89a305efea7fa55a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33405 } } peers { permanent_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39183 } } peers { permanent_uuid: "4bcacb667ac14937b041f7d4694b907b" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 41239 } }
I20260810 22:34:20.623876 26255 leader_election.cc:288] T 5a3330e783344a6c86246dfba1399aab P c33ed31d8faf4bc89a305efea7fa55a9 [CANDIDATE]: Term 1 election: Requested vote from peers bd1ece4b27ae4e6c9c715a0045bac33c (127.16.233.65:33405), 4bcacb667ac14937b041f7d4694b907b (127.16.233.67:41239)
I20260810 22:34:20.624356 25816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a3330e783344a6c86246dfba1399aab" candidate_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c"
I20260810 22:34:20.624524 25816 raft_consensus.cc:3037] T 5a3330e783344a6c86246dfba1399aab P bd1ece4b27ae4e6c9c715a0045bac33c [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:34:20.624549 26123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a3330e783344a6c86246dfba1399aab" candidate_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bcacb667ac14937b041f7d4694b907b"
I20260810 22:34:20.624683 26123 raft_consensus.cc:3037] T 5a3330e783344a6c86246dfba1399aab P 4bcacb667ac14937b041f7d4694b907b [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:34:20.625901 26123 raft_consensus.cc:2454] T 5a3330e783344a6c86246dfba1399aab P 4bcacb667ac14937b041f7d4694b907b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c33ed31d8faf4bc89a305efea7fa55a9 in term 1.
I20260810 22:34:20.625901 25816 raft_consensus.cc:2454] T 5a3330e783344a6c86246dfba1399aab P bd1ece4b27ae4e6c9c715a0045bac33c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c33ed31d8faf4bc89a305efea7fa55a9 in term 1.
I20260810 22:34:20.626258 25894 leader_election.cc:302] T 5a3330e783344a6c86246dfba1399aab P c33ed31d8faf4bc89a305efea7fa55a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd1ece4b27ae4e6c9c715a0045bac33c, c33ed31d8faf4bc89a305efea7fa55a9; no voters:
I20260810 22:34:20.626514 26255 raft_consensus.cc:2781] T 5a3330e783344a6c86246dfba1399aab P c33ed31d8faf4bc89a305efea7fa55a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:34:20.626633 26255 raft_consensus.cc:686] T 5a3330e783344a6c86246dfba1399aab P c33ed31d8faf4bc89a305efea7fa55a9 [term 1 LEADER]: Becoming Leader. State: Replica: c33ed31d8faf4bc89a305efea7fa55a9, State: Running, Role: LEADER
I20260810 22:34:20.626794 26255 consensus_queue.cc:237] T 5a3330e783344a6c86246dfba1399aab P c33ed31d8faf4bc89a305efea7fa55a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33405 } } peers { permanent_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39183 } } peers { permanent_uuid: "4bcacb667ac14937b041f7d4694b907b" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 41239 } }
I20260810 22:34:20.628916 25637 catalog_manager.cc:5495] T 5a3330e783344a6c86246dfba1399aab P c33ed31d8faf4bc89a305efea7fa55a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to c33ed31d8faf4bc89a305efea7fa55a9 (127.16.233.66). New cstate: current_term: 1 leader_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c33ed31d8faf4bc89a305efea7fa55a9" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4bcacb667ac14937b041f7d4694b907b" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 41239 } health_report { overall_health: UNKNOWN } } }
I20260810 22:34:20.651501 26202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:20.647762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34248 (local address 127.16.233.66:39183)
0810 22:34:20.647875 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:20.647880 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:20.647897 (+ 17us) server_negotiation.cc:407] Connection header received
0810 22:34:20.648254 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:20.648260 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:20.648325 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:20.648410 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:34:20.648656 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:20.649320 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:20.650367 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:20.650585 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:20.650678 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:20.651037 (+ 359us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:34:20.651110 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:34:20.651168 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:20.651322 (+ 154us) server_negotiation.cc:299] Negotiation successful
0810 22:34:20.651376 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260810 22:34:20.661736 26202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:20.658548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:33043 (local address 127.16.233.66:39183)
0810 22:34:20.658664 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:20.658670 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:20.658751 (+ 81us) server_negotiation.cc:407] Connection header received
0810 22:34:20.658974 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:20.658978 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:20.659029 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:20.659114 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:34:20.659375 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:20.659880 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:20.660703 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:20.660937 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:20.661050 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:20.661272 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:34:20.661349 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:34:20.661436 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:20.661517 (+ 81us) server_negotiation.cc:299] Negotiation successful
0810 22:34:20.661575 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":9,"server-negotiator.queue_time_us":61}
I20260810 22:34:20.661736 26268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:20.658473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.66:39183 (local address 127.16.233.66:33043)
0810 22:34:20.658664 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:20.658680 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:20.658809 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:20.659121 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:20.659123 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:20.659129 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 22:34:20.659262 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:20.659273 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:20.659992 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:20.659997 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:20.660569 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:20.660577 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:20.661051 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:20.661054 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:20.661132 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:20.661162 (+ 30us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 22:34:20.661437 (+ 275us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 22:34:20.661446 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 22:34:20.661512 (+ 66us) client_negotiation.cc:240] Negotiation successful
0810 22:34:20.661605 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":25}
I20260810 22:34:20.669365 26123 raft_consensus.cc:1261] T 5a3330e783344a6c86246dfba1399aab P 4bcacb667ac14937b041f7d4694b907b [term 1 FOLLOWER]: Refusing update from remote peer c33ed31d8faf4bc89a305efea7fa55a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:34:20.669385 25816 raft_consensus.cc:1261] T 5a3330e783344a6c86246dfba1399aab P bd1ece4b27ae4e6c9c715a0045bac33c [term 1 FOLLOWER]: Refusing update from remote peer c33ed31d8faf4bc89a305efea7fa55a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:34:20.669925 26255 consensus_queue.cc:1035] T 5a3330e783344a6c86246dfba1399aab P c33ed31d8faf4bc89a305efea7fa55a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd1ece4b27ae4e6c9c715a0045bac33c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:34:20.670161 26218 consensus_queue.cc:1035] T 5a3330e783344a6c86246dfba1399aab P c33ed31d8faf4bc89a305efea7fa55a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bcacb667ac14937b041f7d4694b907b" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 41239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:34:20.717446 25636 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:34:20.717653 25636 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:34:20.720358 25636 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 98283e123558435d982c4c04a9e1d6a0: Sending DeleteTablet for 3 replicas of tablet 5a3330e783344a6c86246dfba1399aab
I20260810 22:34:20.721206 26101 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a3330e783344a6c86246dfba1399aab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:34:20 UTC) from {username='oryx', principal='oryx/127.16.233.126@KRBTEST.COM'} at 127.0.0.1:56890
I20260810 22:34:20.721215 25793 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a3330e783344a6c86246dfba1399aab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:34:20 UTC) from {username='oryx', principal='oryx/127.16.233.126@KRBTEST.COM'} at 127.0.0.1:57154
I20260810 22:34:20.721459 26234 tablet_replica.cc:330] T 5a3330e783344a6c86246dfba1399aab P 4bcacb667ac14937b041f7d4694b907b: stopping tablet replica
I20260810 22:34:20.721585 26234 raft_consensus.cc:2229] T 5a3330e783344a6c86246dfba1399aab P 4bcacb667ac14937b041f7d4694b907b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:34:20.721706 26234 raft_consensus.cc:2258] T 5a3330e783344a6c86246dfba1399aab P 4bcacb667ac14937b041f7d4694b907b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:34:20.721865 26233 tablet_replica.cc:330] T 5a3330e783344a6c86246dfba1399aab P bd1ece4b27ae4e6c9c715a0045bac33c: stopping tablet replica
I20260810 22:34:20.721973 26233 raft_consensus.cc:2229] T 5a3330e783344a6c86246dfba1399aab P bd1ece4b27ae4e6c9c715a0045bac33c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:34:20.722090 26233 raft_consensus.cc:2258] T 5a3330e783344a6c86246dfba1399aab P bd1ece4b27ae4e6c9c715a0045bac33c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:34:20.723395 26233 ts_tablet_manager.cc:1905] T 5a3330e783344a6c86246dfba1399aab P bd1ece4b27ae4e6c9c715a0045bac33c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:34:20.726114 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 25720
I20260810 22:34:20.726364 26234 ts_tablet_manager.cc:1905] T 5a3330e783344a6c86246dfba1399aab P 4bcacb667ac14937b041f7d4694b907b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:34:20.730265 25938 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a3330e783344a6c86246dfba1399aab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:34:20 UTC) from {username='oryx', principal='oryx/127.16.233.126@KRBTEST.COM'} at 127.0.0.1:34228
I20260810 22:34:20.730497 26232 tablet_replica.cc:330] T 5a3330e783344a6c86246dfba1399aab P c33ed31d8faf4bc89a305efea7fa55a9: stopping tablet replica
I20260810 22:34:20.730614 26232 raft_consensus.cc:2229] T 5a3330e783344a6c86246dfba1399aab P c33ed31d8faf4bc89a305efea7fa55a9 [term 1 LEADER]: Raft consensus shutting down.
I20260810 22:34:20.731036 26232 raft_consensus.cc:2258] T 5a3330e783344a6c86246dfba1399aab P c33ed31d8faf4bc89a305efea7fa55a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:34:20.732056 26232 ts_tablet_manager.cc:1905] T 5a3330e783344a6c86246dfba1399aab P c33ed31d8faf4bc89a305efea7fa55a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:34:20.732167 26234 ts_tablet_manager.cc:1918] T 5a3330e783344a6c86246dfba1399aab P 4bcacb667ac14937b041f7d4694b907b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 22:34:20.732317 26234 log.cc:1196] T 5a3330e783344a6c86246dfba1399aab P 4bcacb667ac14937b041f7d4694b907b: Deleting WAL directory at /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/ts-2/wal/wals/5a3330e783344a6c86246dfba1399aab
I20260810 22:34:20.732612 26234 ts_tablet_manager.cc:1939] T 5a3330e783344a6c86246dfba1399aab P 4bcacb667ac14937b041f7d4694b907b: Deleting consensus metadata
I20260810 22:34:20.735153 26232 ts_tablet_manager.cc:1918] T 5a3330e783344a6c86246dfba1399aab P c33ed31d8faf4bc89a305efea7fa55a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 22:34:20.735286 26232 log.cc:1196] T 5a3330e783344a6c86246dfba1399aab P c33ed31d8faf4bc89a305efea7fa55a9: Deleting WAL directory at /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786401215402979-17317-0/minicluster-data/ts-1/wal/wals/5a3330e783344a6c86246dfba1399aab
I20260810 22:34:20.735580 26232 ts_tablet_manager.cc:1939] T 5a3330e783344a6c86246dfba1399aab P c33ed31d8faf4bc89a305efea7fa55a9: Deleting consensus metadata
I20260810 22:34:20.736603 25621 catalog_manager.cc:4841] TS 4bcacb667ac14937b041f7d4694b907b (127.16.233.67:41239): tablet 5a3330e783344a6c86246dfba1399aab (table test-table [id=1270c0983d7f46c18fb6869898881700]) successfully deleted
I20260810 22:34:20.736562 25620 catalog_manager.cc:4841] TS c33ed31d8faf4bc89a305efea7fa55a9 (127.16.233.66:39183): tablet 5a3330e783344a6c86246dfba1399aab (table test-table [id=1270c0983d7f46c18fb6869898881700]) successfully deleted
W20260810 22:34:20.741941 25620 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.233.65:33405 (error 108)
I20260810 22:34:20.742096 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 25877
I20260810 22:34:20.745782 26310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:20.744632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.65:33405 (local address 127.0.0.1:57168)
0810 22:34:20.745512 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:20.745623 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.65:33405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":760,"thread_start_us":73,"threads_started":1}
W20260810 22:34:20.745963 25620 catalog_manager.cc:4584] TS bd1ece4b27ae4e6c9c715a0045bac33c (127.16.233.65:33405): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5a3330e783344a6c86246dfba1399aab: Network error: Client connection negotiation failed: client connection to 127.16.233.65:33405: connect: Connection refused (error 111)
I20260810 22:34:20.754473 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 26029
I20260810 22:34:20.764397 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 25602
2026-08-10T22:34:20Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3556 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786401215402979-17317-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:34:20 dist-test-slave-77v9 krb5kdc[26313](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:34:20 dist-test-slave-77v9 krb5kdc[26313](info): set up 2 sockets
Aug 10 22:34:20 dist-test-slave-77v9 krb5kdc[26313](info): commencing operation
krb5kdc: starting...
W20260810 22:34:21.942265 17317 mini_kdc.cc:121] Time spent starting KDC: real 1.149s user 0.002s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Aug 10 22:34:22 dist-test-slave-77v9 krb5kdc[26313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:34:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:34:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.233.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab.
Entry for principal kudu/127.16.233.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab.
I20260810 22:34:22.105880 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786401215402979-17317-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786401215402979-17317-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786401215402979-17317-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:40343
--webserver_interface=127.16.233.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab
--principal=kudu/127.16.233.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:36279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.126:40343
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:34:22.231307 26326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:34:22.231645 26326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:34:22.231693 26326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:34:22.236070 26326 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:34:22.236168 26326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:34:22.236191 26326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:34:22.236223 26326 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:34:22.236243 26326 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:34:22.242690 26326 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:36279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786401215402979-17317-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786401215402979-17317-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.126:40343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:40343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26326
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786401215402979-17317-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:34:22.244122 26326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:34:22.245293 26326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:34:22.255421 26332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:22.255486 26337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:22.255652 26326 server_base.cc:1034] running on GCE node
W20260810 22:34:22.255450 26334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:22.256445 26326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:34:22.257742 26326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:34:22.258996 26326 hybrid_clock.cc:648] HybridClock initialized: now 1786401262258971 us; error 43 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-10T22:34:22Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1495 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:34:22 dist-test-slave-77v9 krb5kdc[26343](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:34:22 dist-test-slave-77v9 krb5kdc[26343](info): set up 2 sockets
Aug 10 22:34:22 dist-test-slave-77v9 krb5kdc[26343](info): commencing operation
krb5kdc: starting...
W20260810 22:34:23.421516 17317 mini_kdc.cc:121] Time spent starting KDC: real 1.132s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Aug 10 22:34:23 dist-test-slave-77v9 krb5kdc[26343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:34:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:34:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.233.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab.
Entry for principal kudu/127.16.233.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab.
I20260810 22:34:23.580257 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:45513
--webserver_interface=127.16.233.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab
--principal=kudu/127.16.233.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:37427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.126:45513
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:34:23.702839 26357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:34:23.703275 26357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:34:23.703361 26357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:34:23.707738 26357 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:34:23.707886 26357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:34:23.707926 26357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:34:23.707945 26357 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:34:23.707962 26357 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:34:23.713771 26357 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:37427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.126:45513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab
--principal=kudu/127.16.233.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:45513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26357
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:34:23.714926 26357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:34:23.715967 26357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:34:23.725739 26363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:23.725726 26365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:23.725745 26362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:23.726140 26357 server_base.cc:1034] running on GCE node
I20260810 22:34:23.726790 26357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:34:23.728124 26357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:34:23.729367 26357 hybrid_clock.cc:648] HybridClock initialized: now 1786401263729335 us; error 43 us; skew 500 ppm
Aug 10 22:34:23 dist-test-slave-77v9 krb5kdc[26343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401263, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:34:23.732596 26357 init.cc:377] Logged in from keytab as kudu/127.16.233.126@KRBTEST.COM (short username kudu)
I20260810 22:34:23.733659 26357 webserver.cc:460] Webserver started at http://127.16.233.126:44919/ using document root <none> and password file <none>
I20260810 22:34:23.734259 26357 fs_manager.cc:359] Metadata directory not provided
I20260810 22:34:23.734357 26357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:34:23.734603 26357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:34:23.736995 26357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/master-0/data/instance:
uuid: "fce4876d3a304282a6b91766c349eaa7"
format_stamp: "Formatted at 2026-08-10 22:34:23 on dist-test-slave-77v9"
server_key: "2ce904e03a3a0f2086486fb03685cfa4"
server_key_iv: "2d031353f34fb2b72301b14c0fea1850"
server_key_version: "encryptionkey@0"
I20260810 22:34:23.737668 26357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/master-0/wal/instance:
uuid: "fce4876d3a304282a6b91766c349eaa7"
format_stamp: "Formatted at 2026-08-10 22:34:23 on dist-test-slave-77v9"
server_key: "2ce904e03a3a0f2086486fb03685cfa4"
server_key_iv: "2d031353f34fb2b72301b14c0fea1850"
server_key_version: "encryptionkey@0"
I20260810 22:34:23.741946 26357 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 22:34:23.744717 26373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:23.746161 26357 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:34:23.746348 26357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/master-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/master-0/wal
uuid: "fce4876d3a304282a6b91766c349eaa7"
format_stamp: "Formatted at 2026-08-10 22:34:23 on dist-test-slave-77v9"
server_key: "2ce904e03a3a0f2086486fb03685cfa4"
server_key_iv: "2d031353f34fb2b72301b14c0fea1850"
server_key_version: "encryptionkey@0"
I20260810 22:34:23.746497 26357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:34:23.769035 26357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:34:23.773097 26357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:34:23.773360 26357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:34:23.782074 26357 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.126:45513
I20260810 22:34:23.782100 26435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.126:45513 every 8 connection(s)
I20260810 22:34:23.783200 26357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
I20260810 22:34:23.786757 26436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:34:23.789562 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 26357
I20260810 22:34:23.789712 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/master-0/wal/instance
I20260810 22:34:23.790096 17317 external_mini_cluster.cc:1428] Setting key 06c32eca1010250aac62459a1cafe58e
I20260810 22:34:23.794095 26436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fce4876d3a304282a6b91766c349eaa7: Bootstrap starting.
I20260810 22:34:23.796700 26436 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fce4876d3a304282a6b91766c349eaa7: Neither blocks nor log segments found. Creating new log.
I20260810 22:34:23.797621 26436 log.cc:824] T 00000000000000000000000000000000 P fce4876d3a304282a6b91766c349eaa7: Log is configured to *not* fsync() on all Append() calls
Aug 10 22:34:23 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:34:23.800372 26436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fce4876d3a304282a6b91766c349eaa7: No bootstrap required, opened a new log
I20260810 22:34:23.803951 26436 raft_consensus.cc:348] T 00000000000000000000000000000000 P fce4876d3a304282a6b91766c349eaa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fce4876d3a304282a6b91766c349eaa7" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 45513 } }
I20260810 22:34:23.804229 26436 raft_consensus.cc:374] T 00000000000000000000000000000000 P fce4876d3a304282a6b91766c349eaa7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:34:23.804294 26436 raft_consensus.cc:729] T 00000000000000000000000000000000 P fce4876d3a304282a6b91766c349eaa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fce4876d3a304282a6b91766c349eaa7, State: Initialized, Role: FOLLOWER
I20260810 22:34:23.804880 26436 consensus_queue.cc:260] T 00000000000000000000000000000000 P fce4876d3a304282a6b91766c349eaa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fce4876d3a304282a6b91766c349eaa7" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 45513 } }
I20260810 22:34:23.805051 26436 raft_consensus.cc:388] T 00000000000000000000000000000000 P fce4876d3a304282a6b91766c349eaa7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:34:23.805142 26436 raft_consensus.cc:482] T 00000000000000000000000000000000 P fce4876d3a304282a6b91766c349eaa7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:34:23.805256 26436 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fce4876d3a304282a6b91766c349eaa7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:34:23.806417 26436 raft_consensus.cc:504] T 00000000000000000000000000000000 P fce4876d3a304282a6b91766c349eaa7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fce4876d3a304282a6b91766c349eaa7" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 45513 } }
I20260810 22:34:23.806540 26439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:23.791733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42564 (local address 127.16.233.126:45513)
0810 22:34:23.792351 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:23.792372 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:23.792412 (+ 40us) server_negotiation.cc:407] Connection header received
0810 22:34:23.793183 (+ 771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:23.793203 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:23.793580 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:23.793943 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:23.794856 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:23.795875 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:23.796773 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:23.797379 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:23.797558 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:23.800725 (+ 3167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:23.800730 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:23.800744 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:23.800773 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:23.803258 (+ 2485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:23.803827 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:23.803839 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:23.803844 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:23.803971 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:23.804351 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:23.804355 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:23.804358 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:23.804858 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:23.805068 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:23.805394 (+ 326us) server_negotiation.cc:299] Negotiation successful
0810 22:34:23.805729 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":179,"threads_started":1}
I20260810 22:34:23.806842 26436 leader_election.cc:302] T 00000000000000000000000000000000 P fce4876d3a304282a6b91766c349eaa7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fce4876d3a304282a6b91766c349eaa7; no voters:
I20260810 22:34:23.807214 26436 leader_election.cc:288] T 00000000000000000000000000000000 P fce4876d3a304282a6b91766c349eaa7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:34:23.807442 26443 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fce4876d3a304282a6b91766c349eaa7 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:34:23.807687 26443 raft_consensus.cc:686] T 00000000000000000000000000000000 P fce4876d3a304282a6b91766c349eaa7 [term 1 LEADER]: Becoming Leader. State: Replica: fce4876d3a304282a6b91766c349eaa7, State: Running, Role: LEADER
I20260810 22:34:23.808043 26443 consensus_queue.cc:237] T 00000000000000000000000000000000 P fce4876d3a304282a6b91766c349eaa7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fce4876d3a304282a6b91766c349eaa7" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 45513 } }
I20260810 22:34:23.809746 26436 sys_catalog.cc:564] T 00000000000000000000000000000000 P fce4876d3a304282a6b91766c349eaa7 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:34:23.809974 26444 sys_catalog.cc:455] T 00000000000000000000000000000000 P fce4876d3a304282a6b91766c349eaa7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fce4876d3a304282a6b91766c349eaa7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fce4876d3a304282a6b91766c349eaa7" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 45513 } } }
I20260810 22:34:23.810108 26444 sys_catalog.cc:458] T 00000000000000000000000000000000 P fce4876d3a304282a6b91766c349eaa7 [sys.catalog]: This master's current role is: LEADER
I20260810 22:34:23.810562 26447 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:34:23.810506 26445 sys_catalog.cc:455] T 00000000000000000000000000000000 P fce4876d3a304282a6b91766c349eaa7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fce4876d3a304282a6b91766c349eaa7. Latest consensus state: current_term: 1 leader_uuid: "fce4876d3a304282a6b91766c349eaa7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fce4876d3a304282a6b91766c349eaa7" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 45513 } } }
I20260810 22:34:23.811201 26445 sys_catalog.cc:458] T 00000000000000000000000000000000 P fce4876d3a304282a6b91766c349eaa7 [sys.catalog]: This master's current role is: LEADER
I20260810 22:34:23.814177 26447 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:34:23.820709 26447 catalog_manager.cc:1347] Generated new cluster ID: fd5bab37f4724e4cb14e6dd562a8b51b
I20260810 22:34:23.820824 26447 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:34:23.843904 26447 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:34:23.845525 26447 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:34:23.851226 26447 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fce4876d3a304282a6b91766c349eaa7: Generated new TSK 0
I20260810 22:34:23.852016 26447 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.233.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab.
Entry for principal kudu/127.16.233.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab.
I20260810 22:34:23.888633 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.65:0
--local_ip_for_outbound_sockets=127.16.233.65
--webserver_interface=127.16.233.65
--webserver_port=0
--tserver_master_addrs=127.16.233.126:45513
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab
--principal=kudu/127.16.233.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:37427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:34:24.013065 26469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:34:24.013404 26469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:34:24.013487 26469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:34:24.017992 26469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:34:24.018092 26469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:34:24.018159 26469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.65
I20260810 22:34:24.024290 26469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:37427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab
--principal=kudu/127.16.233.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.233.65
--webserver_port=0
--tserver_master_addrs=127.16.233.126:45513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.65
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:34:24.025666 26469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:34:24.026859 26469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:34:24.037590 26474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:24.037616 26477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:24.037596 26475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:24.038641 26469 server_base.cc:1034] running on GCE node
I20260810 22:34:24.039232 26469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:34:24.039965 26469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:34:24.041141 26469 hybrid_clock.cc:648] HybridClock initialized: now 1786401264041094 us; error 66 us; skew 500 ppm
Aug 10 22:34:24 dist-test-slave-77v9 krb5kdc[26343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401264, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:34:24.044579 26469 init.cc:377] Logged in from keytab as kudu/127.16.233.65@KRBTEST.COM (short username kudu)
I20260810 22:34:24.045720 26469 webserver.cc:460] Webserver started at http://127.16.233.65:38455/ using document root <none> and password file <none>
I20260810 22:34:24.046350 26469 fs_manager.cc:359] Metadata directory not provided
I20260810 22:34:24.046461 26469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:34:24.046710 26469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:34:24.049068 26469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-0/data/instance:
uuid: "e44552b4bfab46f59ffe6a9d5956637c"
format_stamp: "Formatted at 2026-08-10 22:34:24 on dist-test-slave-77v9"
server_key: "f40a01781a3969882057d84cd7707afe"
server_key_iv: "bfeb54aac959b510459531fc129958c5"
server_key_version: "encryptionkey@0"
I20260810 22:34:24.049685 26469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-0/wal/instance:
uuid: "e44552b4bfab46f59ffe6a9d5956637c"
format_stamp: "Formatted at 2026-08-10 22:34:24 on dist-test-slave-77v9"
server_key: "f40a01781a3969882057d84cd7707afe"
server_key_iv: "bfeb54aac959b510459531fc129958c5"
server_key_version: "encryptionkey@0"
I20260810 22:34:24.054059 26469 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 22:34:24.057062 26484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:24.058571 26469 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 22:34:24.058739 26469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-0/wal
uuid: "e44552b4bfab46f59ffe6a9d5956637c"
format_stamp: "Formatted at 2026-08-10 22:34:24 on dist-test-slave-77v9"
server_key: "f40a01781a3969882057d84cd7707afe"
server_key_iv: "bfeb54aac959b510459531fc129958c5"
server_key_version: "encryptionkey@0"
I20260810 22:34:24.058885 26469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:34:24.081408 26469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:34:24.085228 26469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:34:24.085487 26469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:34:24.086308 26469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:34:24.087306 26469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:34:24.087383 26469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:24.087477 26469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:34:24.087520 26469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:24.097723 26469 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.65:32773
I20260810 22:34:24.097780 26621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.65:32773 every 8 connection(s)
I20260810 22:34:24.098901 26469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
I20260810 22:34:24.108001 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 26469
I20260810 22:34:24.108158 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-0/wal/instance
I20260810 22:34:24.108494 17317 external_mini_cluster.cc:1428] Setting key de202b52301343a20a7df266fd5a50d4
Aug 10 22:34:24 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401264, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.65@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:34:24.114902 26439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:24.101011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:57895 (local address 127.16.233.126:45513)
0810 22:34:24.101186 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:24.101190 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:24.102305 (+ 1115us) server_negotiation.cc:407] Connection header received
0810 22:34:24.103201 (+ 896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:24.103204 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:24.103256 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:24.103341 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:24.104620 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:24.105311 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:24.106585 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:24.106875 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:24.106974 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:24.110163 (+ 3189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:24.110168 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:24.110172 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:24.110205 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:24.112118 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:24.112865 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:24.112871 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:24.112876 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:24.112989 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:24.113508 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:24.113513 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:24.113516 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:24.113805 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:24.113919 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:24.114479 (+ 560us) server_negotiation.cc:299] Negotiation successful
0810 22:34:24.114649 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 22:34:24.115865 26625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:24.101380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:45513 (local address 127.16.233.65:57895)
0810 22:34:24.102104 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:24.102150 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:24.102983 (+ 833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:24.103506 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:24.103513 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:24.103945 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:24.104454 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:24.104469 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:24.105454 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:24.105457 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:24.106423 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:24.106433 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:24.107061 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:24.107064 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:24.107224 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:24.108037 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:24.108054 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:24.109970 (+ 1916us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:34:24.112285 (+ 2315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:24.112296 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:24.112320 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:24.112738 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:24.113187 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:24.113190 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:24.113192 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:24.113365 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:24.113928 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:34:24.113934 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:34:24.114206 (+ 272us) client_negotiation.cc:769] Sending connection context
0810 22:34:24.114411 (+ 205us) client_negotiation.cc:240] Negotiation successful
0810 22:34:24.114729 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":124,"threads_started":1}
I20260810 22:34:24.117417 26622 heartbeater.cc:347] Connected to a master server at 127.16.233.126:45513
I20260810 22:34:24.117688 26622 heartbeater.cc:464] Registering TS with master...
I20260810 22:34:24.118237 26622 heartbeater.cc:511] Master 127.16.233.126:45513 requested a full tablet report, sending...
I20260810 22:34:24.120198 26392 ts_manager.cc:194] Registered new tserver with Master: e44552b4bfab46f59ffe6a9d5956637c (127.16.233.65:32773)
WARNING: no policy specified for kudu/127.16.233.66@KRBTEST.COM; defaulting to no policy
I20260810 22:34:24.121826 26392 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.65@KRBTEST.COM'} at 127.16.233.65:57895
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab.
Entry for principal kudu/127.16.233.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab.
I20260810 22:34:24.144078 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.66:0
--local_ip_for_outbound_sockets=127.16.233.66
--webserver_interface=127.16.233.66
--webserver_port=0
--tserver_master_addrs=127.16.233.126:45513
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab
--principal=kudu/127.16.233.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:37427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:34:24.264726 26628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:34:24.265100 26628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:34:24.265151 26628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:34:24.269886 26628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:34:24.269976 26628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:34:24.270042 26628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.66
I20260810 22:34:24.276049 26628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:37427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab
--principal=kudu/127.16.233.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.233.66
--webserver_port=0
--tserver_master_addrs=127.16.233.126:45513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.66
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:34:24.277354 26628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:34:24.278486 26628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:34:24.289428 26634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:24.289469 26628 server_base.cc:1034] running on GCE node
W20260810 22:34:24.289495 26636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:24.289428 26633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:24.290298 26628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:34:24.291039 26628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:34:24.292239 26628 hybrid_clock.cc:648] HybridClock initialized: now 1786401264292197 us; error 58 us; skew 500 ppm
Aug 10 22:34:24 dist-test-slave-77v9 krb5kdc[26343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401264, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:34:24.295658 26628 init.cc:377] Logged in from keytab as kudu/127.16.233.66@KRBTEST.COM (short username kudu)
I20260810 22:34:24.296689 26628 webserver.cc:460] Webserver started at http://127.16.233.66:39557/ using document root <none> and password file <none>
I20260810 22:34:24.297638 26628 fs_manager.cc:359] Metadata directory not provided
I20260810 22:34:24.297755 26628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:34:24.298059 26628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:34:24.302126 26628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-1/data/instance:
uuid: "9a1453a3b85b41069792018e50254f9e"
format_stamp: "Formatted at 2026-08-10 22:34:24 on dist-test-slave-77v9"
server_key: "97304a048e3d1baf28afcc1e4b84a33f"
server_key_iv: "f1d78cf23dae479aa94766e796729c86"
server_key_version: "encryptionkey@0"
I20260810 22:34:24.303179 26628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-1/wal/instance:
uuid: "9a1453a3b85b41069792018e50254f9e"
format_stamp: "Formatted at 2026-08-10 22:34:24 on dist-test-slave-77v9"
server_key: "97304a048e3d1baf28afcc1e4b84a33f"
server_key_iv: "f1d78cf23dae479aa94766e796729c86"
server_key_version: "encryptionkey@0"
I20260810 22:34:24.309686 26628 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260810 22:34:24.314035 26645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:24.315970 26628 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260810 22:34:24.316162 26628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-1/wal
uuid: "9a1453a3b85b41069792018e50254f9e"
format_stamp: "Formatted at 2026-08-10 22:34:24 on dist-test-slave-77v9"
server_key: "97304a048e3d1baf28afcc1e4b84a33f"
server_key_iv: "f1d78cf23dae479aa94766e796729c86"
server_key_version: "encryptionkey@0"
I20260810 22:34:24.316421 26628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:34:24.371685 26628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:34:24.376574 26628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:34:24.376876 26628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:34:24.377852 26628 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:34:24.379272 26628 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:34:24.379346 26628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:24.379416 26628 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:34:24.379457 26628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:24.395020 26628 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.66:35243
I20260810 22:34:24.395030 26779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.66:35243 every 8 connection(s)
I20260810 22:34:24.396685 26628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
I20260810 22:34:24.404112 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 26628
I20260810 22:34:24.404295 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-1/wal/instance
I20260810 22:34:24.404706 17317 external_mini_cluster.cc:1428] Setting key bd1a602ea41731850285e63461ae8915
Aug 10 22:34:24 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401264, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.66@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:34:24.416683 26439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:24.399202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:34397 (local address 127.16.233.126:45513)
0810 22:34:24.399356 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:24.399360 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:24.401107 (+ 1747us) server_negotiation.cc:407] Connection header received
0810 22:34:24.402171 (+ 1064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:24.402182 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:24.402255 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:24.402369 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:24.403805 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:24.404490 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:24.406152 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:24.406711 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:24.406885 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:24.410684 (+ 3799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:24.410689 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:24.410692 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:24.410726 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:24.413301 (+ 2575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:24.414195 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:24.414199 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:24.414202 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:24.414321 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:24.414889 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:24.414895 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:24.414900 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:24.415227 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:24.415342 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:24.416271 (+ 929us) server_negotiation.cc:299] Negotiation successful
0810 22:34:24.416458 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 22:34:24.418789 26782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:24.399864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:45513 (local address 127.16.233.66:34397)
0810 22:34:24.400920 (+ 1056us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:24.400978 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:24.401919 (+ 941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:24.402575 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:24.402582 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:24.403040 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:24.403598 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:24.403612 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:24.404636 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:24.404645 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:24.405993 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:24.406007 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:24.406927 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:24.406932 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:24.407195 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:24.408080 (+ 885us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:24.408109 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:24.410416 (+ 2307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:34:24.413488 (+ 3072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:24.413500 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:24.413526 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:24.414031 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:24.414468 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:24.414475 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:24.414480 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:24.414724 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:24.415436 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:34:24.415447 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:34:24.415911 (+ 464us) client_negotiation.cc:769] Sending connection context
0810 22:34:24.416957 (+ 1046us) client_negotiation.cc:240] Negotiation successful
0810 22:34:24.417437 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700,"thread_start_us":229,"threads_started":1}
WARNING: no policy specified for kudu/127.16.233.67@KRBTEST.COM; defaulting to no policy
I20260810 22:34:24.420733 26780 heartbeater.cc:347] Connected to a master server at 127.16.233.126:45513
I20260810 22:34:24.421087 26780 heartbeater.cc:464] Registering TS with master...
I20260810 22:34:24.421809 26780 heartbeater.cc:511] Master 127.16.233.126:45513 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.67@KRBTEST.COM" created.
I20260810 22:34:24.423525 26392 ts_manager.cc:194] Registered new tserver with Master: 9a1453a3b85b41069792018e50254f9e (127.16.233.66:35243)
I20260810 22:34:24.424242 26392 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.66@KRBTEST.COM'} at 127.16.233.66:34397
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab.
Entry for principal kudu/127.16.233.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab.
I20260810 22:34:24.445807 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.67:0
--local_ip_for_outbound_sockets=127.16.233.67
--webserver_interface=127.16.233.67
--webserver_port=0
--tserver_master_addrs=127.16.233.126:45513
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab
--principal=kudu/127.16.233.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:37427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:34:24.572739 26786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:34:24.573244 26786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:34:24.573369 26786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:34:24.578929 26786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:34:24.579067 26786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:34:24.579203 26786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.67
I20260810 22:34:24.585891 26786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:37427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab
--principal=kudu/127.16.233.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.233.67
--webserver_port=0
--tserver_master_addrs=127.16.233.126:45513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.67
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:34:24.587273 26786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:34:24.588496 26786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:34:24.599521 26795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:24.599715 26786 server_base.cc:1034] running on GCE node
W20260810 22:34:24.599532 26793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:24.599586 26792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:24.600448 26786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:34:24.601239 26786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:34:24.602466 26786 hybrid_clock.cc:648] HybridClock initialized: now 1786401264602434 us; error 41 us; skew 500 ppm
Aug 10 22:34:24 dist-test-slave-77v9 krb5kdc[26343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401264, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:34:24.605792 26786 init.cc:377] Logged in from keytab as kudu/127.16.233.67@KRBTEST.COM (short username kudu)
I20260810 22:34:24.606943 26786 webserver.cc:460] Webserver started at http://127.16.233.67:39573/ using document root <none> and password file <none>
I20260810 22:34:24.607676 26786 fs_manager.cc:359] Metadata directory not provided
I20260810 22:34:24.607793 26786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:34:24.608088 26786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:34:24.610842 26786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-2/data/instance:
uuid: "c33b3eff30b14037b3c6552d562a89c5"
format_stamp: "Formatted at 2026-08-10 22:34:24 on dist-test-slave-77v9"
server_key: "c4277767bb40cb6752a7f6446c2fc430"
server_key_iv: "be28400aca87e98b914830098bdb54ba"
server_key_version: "encryptionkey@0"
I20260810 22:34:24.611625 26786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-2/wal/instance:
uuid: "c33b3eff30b14037b3c6552d562a89c5"
format_stamp: "Formatted at 2026-08-10 22:34:24 on dist-test-slave-77v9"
server_key: "c4277767bb40cb6752a7f6446c2fc430"
server_key_iv: "be28400aca87e98b914830098bdb54ba"
server_key_version: "encryptionkey@0"
I20260810 22:34:24.616371 26786 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260810 22:34:24.619531 26803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:24.621461 26786 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:34:24.621625 26786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-2/wal
uuid: "c33b3eff30b14037b3c6552d562a89c5"
format_stamp: "Formatted at 2026-08-10 22:34:24 on dist-test-slave-77v9"
server_key: "c4277767bb40cb6752a7f6446c2fc430"
server_key_iv: "be28400aca87e98b914830098bdb54ba"
server_key_version: "encryptionkey@0"
I20260810 22:34:24.621762 26786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:34:24.642328 26786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:34:24.646090 26786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:34:24.646335 26786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:34:24.646997 26786 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:34:24.647886 26786 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:34:24.647934 26786 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:24.647977 26786 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:34:24.648031 26786 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:24.658018 26786 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.67:38497
I20260810 22:34:24.658057 26931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.67:38497 every 8 connection(s)
I20260810 22:34:24.659067 26786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
I20260810 22:34:24.665650 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 26786
I20260810 22:34:24.665849 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-2/wal/instance
I20260810 22:34:24.666188 17317 external_mini_cluster.cc:1428] Setting key ee0d5d4d916ae14d788ddc6e4605ee1a
Aug 10 22:34:24 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401264, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.67@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:34:24.676052 26439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:24.661215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:56329 (local address 127.16.233.126:45513)
0810 22:34:24.661355 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:24.661364 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:24.662475 (+ 1111us) server_negotiation.cc:407] Connection header received
0810 22:34:24.664103 (+ 1628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:24.664108 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:24.664182 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:24.664298 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:24.665763 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:24.666549 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:24.668350 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:24.668730 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:24.668970 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:24.672012 (+ 3042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:24.672016 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:24.672018 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:24.672045 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:24.673894 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:24.674498 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:24.674502 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:24.674503 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:24.674591 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:24.674925 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:24.674927 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:24.674929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:24.675133 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:24.675215 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:24.675743 (+ 528us) server_negotiation.cc:299] Negotiation successful
0810 22:34:24.675919 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 22:34:24.677027 26934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:24.661593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:45513 (local address 127.16.233.67:56329)
0810 22:34:24.662178 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:24.662276 (+ 98us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:24.663840 (+ 1564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:24.664501 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:24.664509 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:24.665020 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:24.665582 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:24.665598 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:24.666711 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:24.666715 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:24.668201 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:24.668213 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:24.669039 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:24.669043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:24.669221 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:24.669951 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:24.669972 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:24.671777 (+ 1805us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:34:24.674040 (+ 2263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:24.674046 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:24.674062 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:24.674377 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:24.674678 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:24.674680 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:24.674681 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:24.674831 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:24.675221 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:34:24.675226 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:34:24.675501 (+ 275us) client_negotiation.cc:769] Sending connection context
0810 22:34:24.675667 (+ 166us) client_negotiation.cc:240] Negotiation successful
0810 22:34:24.675970 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":138,"threads_started":1}
I20260810 22:34:24.678570 26932 heartbeater.cc:347] Connected to a master server at 127.16.233.126:45513
I20260810 22:34:24.678898 26932 heartbeater.cc:464] Registering TS with master...
I20260810 22:34:24.679548 26932 heartbeater.cc:511] Master 127.16.233.126:45513 requested a full tablet report, sending...
I20260810 22:34:24.680868 26394 ts_manager.cc:194] Registered new tserver with Master: c33b3eff30b14037b3c6552d562a89c5 (127.16.233.67:38497)
I20260810 22:34:24.681542 26394 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.67@KRBTEST.COM'} at 127.16.233.67:56329
I20260810 22:34:24.693406 17317 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.16.233.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.233.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.233.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/krb5kdc/oryx_127.16.233.65.keytab.
Entry for principal oryx/127.16.233.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/krb5kdc/oryx_127.16.233.65.keytab.
WARNING: no policy specified for oryx/127.16.233.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.233.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.233.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/krb5kdc/oryx_127.16.233.66.keytab.
Entry for principal oryx/127.16.233.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/krb5kdc/oryx_127.16.233.66.keytab.
WARNING: no policy specified for oryx/127.16.233.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.233.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.233.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/krb5kdc/oryx_127.16.233.67.keytab.
Entry for principal oryx/127.16.233.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/krb5kdc/oryx_127.16.233.67.keytab.
I20260810 22:34:24.793169 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 26469
I20260810 22:34:24.805696 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 26628
I20260810 22:34:24.813867 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 26786
I20260810 22:34:24.822549 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 26357
I20260810 22:34:24.831382 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:45513
--webserver_interface=127.16.233.126
--webserver_port=44919
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab
--principal=kudu/127.16.233.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:37427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.126:45513
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:34:24.950724 26943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:34:24.951097 26943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:34:24.951172 26943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:34:24.956099 26943 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:34:24.956197 26943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:34:24.956236 26943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:34:24.956275 26943 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:34:24.956312 26943 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:34:24.962070 26943 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:37427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.126:45513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab
--principal=kudu/127.16.233.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:45513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.126
--webserver_port=44919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:34:24.963320 26943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:34:24.964452 26943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:34:24.973695 26948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:24.973663 26949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:24.973701 26952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:24.973881 26943 server_base.cc:1034] running on GCE node
I20260810 22:34:24.974712 26943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:34:24.976068 26943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:34:24.977257 26943 hybrid_clock.cc:648] HybridClock initialized: now 1786401264977218 us; error 52 us; skew 500 ppm
Aug 10 22:34:24 dist-test-slave-77v9 krb5kdc[26343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401264, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:34:24.980559 26943 init.cc:377] Logged in from keytab as kudu/127.16.233.126@KRBTEST.COM (short username kudu)
I20260810 22:34:24.981644 26943 webserver.cc:460] Webserver started at http://127.16.233.126:44919/ using document root <none> and password file <none>
I20260810 22:34:24.982218 26943 fs_manager.cc:359] Metadata directory not provided
I20260810 22:34:24.982321 26943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:34:24.986727 26943 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260810 22:34:24.989207 26959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:24.990626 26943 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 22:34:24.990810 26943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/master-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/master-0/wal
uuid: "fce4876d3a304282a6b91766c349eaa7"
format_stamp: "Formatted at 2026-08-10 22:34:23 on dist-test-slave-77v9"
server_key: "2ce904e03a3a0f2086486fb03685cfa4"
server_key_iv: "2d031353f34fb2b72301b14c0fea1850"
server_key_version: "encryptionkey@0"
I20260810 22:34:24.991421 26943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:34:25.009714 26943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:34:25.013343 26943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:34:25.013582 26943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:34:25.021709 26943 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.126:45513
I20260810 22:34:25.021723 27019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.126:45513 every 8 connection(s)
I20260810 22:34:25.022954 26943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
I20260810 22:34:25.027842 27020 sys_catalog.cc:263] Verifying existing consensus state
I20260810 22:34:25.029837 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 26943
I20260810 22:34:25.029829 27020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fce4876d3a304282a6b91766c349eaa7: Bootstrap starting.
I20260810 22:34:25.030419 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.65:32773
--local_ip_for_outbound_sockets=127.16.233.65
--tserver_master_addrs=127.16.233.126:45513
--webserver_port=38455
--webserver_interface=127.16.233.65
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab
--principal=kudu/127.16.233.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:37427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.16.233.65
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/krb5kdc/oryx_127.16.233.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 22:34:25.048977 27020 log.cc:824] T 00000000000000000000000000000000 P fce4876d3a304282a6b91766c349eaa7: Log is configured to *not* fsync() on all Append() calls
I20260810 22:34:25.056116 27020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fce4876d3a304282a6b91766c349eaa7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 22:34:25.056552 27020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fce4876d3a304282a6b91766c349eaa7: Bootstrap complete.
I20260810 22:34:25.061236 27020 raft_consensus.cc:348] T 00000000000000000000000000000000 P fce4876d3a304282a6b91766c349eaa7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fce4876d3a304282a6b91766c349eaa7" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 45513 } }
I20260810 22:34:25.061748 27020 raft_consensus.cc:729] T 00000000000000000000000000000000 P fce4876d3a304282a6b91766c349eaa7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fce4876d3a304282a6b91766c349eaa7, State: Initialized, Role: FOLLOWER
I20260810 22:34:25.062274 27020 consensus_queue.cc:260] T 00000000000000000000000000000000 P fce4876d3a304282a6b91766c349eaa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fce4876d3a304282a6b91766c349eaa7" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 45513 } }
I20260810 22:34:25.062458 27020 raft_consensus.cc:388] T 00000000000000000000000000000000 P fce4876d3a304282a6b91766c349eaa7 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:34:25.062548 27020 raft_consensus.cc:482] T 00000000000000000000000000000000 P fce4876d3a304282a6b91766c349eaa7 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:34:25.062676 27020 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fce4876d3a304282a6b91766c349eaa7 [term 1 FOLLOWER]: Advancing to term 2
I20260810 22:34:25.064971 27020 raft_consensus.cc:504] T 00000000000000000000000000000000 P fce4876d3a304282a6b91766c349eaa7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fce4876d3a304282a6b91766c349eaa7" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 45513 } }
I20260810 22:34:25.065332 27020 leader_election.cc:302] T 00000000000000000000000000000000 P fce4876d3a304282a6b91766c349eaa7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fce4876d3a304282a6b91766c349eaa7; no voters:
I20260810 22:34:25.065716 27020 leader_election.cc:288] T 00000000000000000000000000000000 P fce4876d3a304282a6b91766c349eaa7 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 22:34:25.065819 27025 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fce4876d3a304282a6b91766c349eaa7 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 22:34:25.066115 27025 raft_consensus.cc:686] T 00000000000000000000000000000000 P fce4876d3a304282a6b91766c349eaa7 [term 2 LEADER]: Becoming Leader. State: Replica: fce4876d3a304282a6b91766c349eaa7, State: Running, Role: LEADER
I20260810 22:34:25.066437 27025 consensus_queue.cc:237] T 00000000000000000000000000000000 P fce4876d3a304282a6b91766c349eaa7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fce4876d3a304282a6b91766c349eaa7" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 45513 } }
I20260810 22:34:25.067112 27020 sys_catalog.cc:564] T 00000000000000000000000000000000 P fce4876d3a304282a6b91766c349eaa7 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:34:25.068080 27027 sys_catalog.cc:455] T 00000000000000000000000000000000 P fce4876d3a304282a6b91766c349eaa7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fce4876d3a304282a6b91766c349eaa7. Latest consensus state: current_term: 2 leader_uuid: "fce4876d3a304282a6b91766c349eaa7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fce4876d3a304282a6b91766c349eaa7" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 45513 } } }
I20260810 22:34:25.068243 27027 sys_catalog.cc:458] T 00000000000000000000000000000000 P fce4876d3a304282a6b91766c349eaa7 [sys.catalog]: This master's current role is: LEADER
I20260810 22:34:25.068534 27026 sys_catalog.cc:455] T 00000000000000000000000000000000 P fce4876d3a304282a6b91766c349eaa7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "fce4876d3a304282a6b91766c349eaa7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fce4876d3a304282a6b91766c349eaa7" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 45513 } } }
I20260810 22:34:25.068620 27026 sys_catalog.cc:458] T 00000000000000000000000000000000 P fce4876d3a304282a6b91766c349eaa7 [sys.catalog]: This master's current role is: LEADER
I20260810 22:34:25.068660 27036 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:34:25.072276 27036 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:34:25.072999 27036 catalog_manager.cc:1259] Loaded cluster ID: fd5bab37f4724e4cb14e6dd562a8b51b
I20260810 22:34:25.073071 27036 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:34:25.075503 27036 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:34:25.076272 27036 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fce4876d3a304282a6b91766c349eaa7: Loaded TSK: 0
I20260810 22:34:25.077601 27036 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 22:34:25.153484 27023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:34:25.153889 27023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:34:25.153990 27023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:34:25.158138 27023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:34:25.158238 27023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:34:25.158347 27023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.65
I20260810 22:34:25.163954 27023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:37427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/krb5kdc/oryx_127.16.233.65.keytab
--principal=oryx/127.16.233.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.65:32773
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.233.65
--webserver_port=38455
--tserver_master_addrs=127.16.233.126:45513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.65
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:34:25.165176 27023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:34:25.166324 27023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:34:25.177837 27052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:25.177841 27051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:25.177985 27055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:25.178380 27023 server_base.cc:1034] running on GCE node
I20260810 22:34:25.178997 27023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:34:25.179872 27023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:34:25.181097 27023 hybrid_clock.cc:648] HybridClock initialized: now 1786401265181076 us; error 29 us; skew 500 ppm
Aug 10 22:34:25 dist-test-slave-77v9 krb5kdc[26343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401265, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:34:25.185091 27023 init.cc:377] Logged in from keytab as oryx/127.16.233.65@KRBTEST.COM (short username oryx)
I20260810 22:34:25.186358 27023 webserver.cc:460] Webserver started at http://127.16.233.65:38455/ using document root <none> and password file <none>
I20260810 22:34:25.187165 27023 fs_manager.cc:359] Metadata directory not provided
I20260810 22:34:25.187285 27023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:34:25.192909 27023 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 22:34:25.195904 27065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:25.197647 27023 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:34:25.197790 27023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-0/wal
uuid: "e44552b4bfab46f59ffe6a9d5956637c"
format_stamp: "Formatted at 2026-08-10 22:34:24 on dist-test-slave-77v9"
server_key: "f40a01781a3969882057d84cd7707afe"
server_key_iv: "bfeb54aac959b510459531fc129958c5"
server_key_version: "encryptionkey@0"
I20260810 22:34:25.198525 27023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:34:25.232688 27023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:34:25.236479 27023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:34:25.236712 27023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:34:25.237390 27023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:34:25.238302 27023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:34:25.238371 27023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:25.238444 27023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:34:25.238484 27023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:25.249069 27023 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.65:32773
I20260810 22:34:25.249092 27192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.65:32773 every 8 connection(s)
I20260810 22:34:25.250224 27023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
I20260810 22:34:25.259714 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 27023
I20260810 22:34:25.260280 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.66:35243
--local_ip_for_outbound_sockets=127.16.233.66
--tserver_master_addrs=127.16.233.126:45513
--webserver_port=39557
--webserver_interface=127.16.233.66
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab
--principal=kudu/127.16.233.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:37427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.16.233.66
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/krb5kdc/oryx_127.16.233.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 22:34:25 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.65@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:34:25 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.65@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:34:25.264403 27195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:25.252632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:45513 (local address 127.16.233.65:48445)
0810 22:34:25.253445 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:25.253501 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:25.254362 (+ 861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:25.255500 (+ 1138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:25.255508 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:25.255982 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:25.256468 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:25.256484 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:25.257901 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:25.257905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:25.258789 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:25.258799 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:25.259638 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:25.259642 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:25.259845 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:25.260639 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:25.260657 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:25.263554 (+ 2897us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.126:45513: Server oryx/127.16.233.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":186,"threads_started":1}
W20260810 22:34:25.264912 27196 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:34:25.252520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:48445 (local address 127.16.233.126:45513)
0810 22:34:25.253191 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:25.253214 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:25.253627 (+ 413us) server_negotiation.cc:407] Connection header received
0810 22:34:25.254784 (+ 1157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:25.254794 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:25.255091 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:25.255430 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:25.256653 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:25.257729 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:25.258941 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:25.259447 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:25.259627 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:25.263919 (+ 4292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.65:48445: BlockingRecv error: recv got EOF from 127.16.233.65:48445 (error 108)
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":247,"threads_started":1}
W20260810 22:34:25.265043 27193 heartbeater.cc:644] Failed to heartbeat to 127.16.233.126:45513 (0 consecutive failures): Not authorized: Failed to ping master at 127.16.233.126:45513: Client connection negotiation failed: client connection to 127.16.233.126:45513: Server oryx/127.16.233.126@KRBTEST.COM not found in Kerberos database
Aug 10 22:34:25 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.65@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:34:25 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.65@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:34:25.272142 27195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:25.265558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:45513 (local address 127.16.233.65:40677)
0810 22:34:25.265709 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:25.265729 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:25.265828 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:25.266145 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:25.266151 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:25.266397 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:25.266578 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:25.266590 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:25.267455 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:25.267462 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:25.268530 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:25.268545 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:25.269116 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:25.269122 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:25.269258 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:25.269793 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:25.269812 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:25.271964 (+ 2152us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.126:45513: Server oryx/127.16.233.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":57}
W20260810 22:34:25.272204 27196 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:34:25.265683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:40677 (local address 127.16.233.126:45513)
0810 22:34:25.265801 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:25.265804 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:25.265819 (+ 15us) server_negotiation.cc:407] Connection header received
0810 22:34:25.265972 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:25.265975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:25.266021 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:25.266093 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:25.266732 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:25.267321 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:25.268677 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:25.268985 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:25.269075 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:25.272088 (+ 3013us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.65:40677: BlockingRecv error: recv got EOF from 127.16.233.65:40677 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Aug 10 22:34:25 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.65@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:34:25 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.65@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:34:25.278980 27195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:25.272862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:45513 (local address 127.16.233.65:43609)
0810 22:34:25.273036 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:25.273065 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:25.273146 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:25.273404 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:25.273407 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:25.273547 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:25.273626 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:25.273630 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:25.274794 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:25.274798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:25.275357 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:25.275365 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:25.275967 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:25.275970 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:25.276042 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:25.276573 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:25.276585 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:25.278828 (+ 2243us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.126:45513: Server oryx/127.16.233.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":82}
W20260810 22:34:25.279188 27196 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:34:25.272986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:43609 (local address 127.16.233.126:45513)
0810 22:34:25.273131 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:25.273138 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:25.273159 (+ 21us) server_negotiation.cc:407] Connection header received
0810 22:34:25.273215 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:25.273220 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:25.273284 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:25.273400 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:25.273743 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:25.274641 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:25.275475 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:25.275833 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:25.275940 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:25.279011 (+ 3071us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.65:43609: BlockingRecv error: recv got EOF from 127.16.233.65:43609 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260810 22:34:25.279248 27193 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 22:34:25.382788 27197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:34:25.383132 27197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:34:25.383208 27197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:34:25.387444 27197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:34:25.387553 27197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:34:25.387650 27197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.66
I20260810 22:34:25.393462 27197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:37427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/krb5kdc/oryx_127.16.233.66.keytab
--principal=oryx/127.16.233.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.66:35243
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.233.66
--webserver_port=39557
--tserver_master_addrs=127.16.233.126:45513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.66
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:34:25.394721 27197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:34:25.395859 27197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:34:25.406284 27206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:25.406400 27204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:25.406440 27202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:25.406687 27197 server_base.cc:1034] running on GCE node
I20260810 22:34:25.407092 27197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:34:25.407720 27197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:34:25.408887 27197 hybrid_clock.cc:648] HybridClock initialized: now 1786401265408852 us; error 45 us; skew 500 ppm
Aug 10 22:34:25 dist-test-slave-77v9 krb5kdc[26343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401265, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:34:25.411859 27197 init.cc:377] Logged in from keytab as oryx/127.16.233.66@KRBTEST.COM (short username oryx)
I20260810 22:34:25.412911 27197 webserver.cc:460] Webserver started at http://127.16.233.66:39557/ using document root <none> and password file <none>
I20260810 22:34:25.413509 27197 fs_manager.cc:359] Metadata directory not provided
I20260810 22:34:25.413592 27197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:34:25.417992 27197 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 22:34:25.420244 27213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:25.421623 27197 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 22:34:25.421752 27197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-1/wal
uuid: "9a1453a3b85b41069792018e50254f9e"
format_stamp: "Formatted at 2026-08-10 22:34:24 on dist-test-slave-77v9"
server_key: "97304a048e3d1baf28afcc1e4b84a33f"
server_key_iv: "f1d78cf23dae479aa94766e796729c86"
server_key_version: "encryptionkey@0"
I20260810 22:34:25.422310 27197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:34:25.438498 27197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:34:25.441799 27197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:34:25.442015 27197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:34:25.442662 27197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:34:25.443536 27197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:34:25.443588 27197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:25.443647 27197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:34:25.443681 27197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:25.453138 27197 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.66:35243
I20260810 22:34:25.453159 27350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.66:35243 every 8 connection(s)
I20260810 22:34:25.454247 27197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
I20260810 22:34:25.460641 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 27197
I20260810 22:34:25.461301 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.67:38497
--local_ip_for_outbound_sockets=127.16.233.67
--tserver_master_addrs=127.16.233.126:45513
--webserver_port=39573
--webserver_interface=127.16.233.67
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab
--principal=kudu/127.16.233.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:37427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.16.233.67
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/krb5kdc/oryx_127.16.233.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 22:34:25 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.66@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:34:25 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.66@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
W20260810 22:34:25.467622 27196 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:34:25.456228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:57811 (local address 127.16.233.126:45513)
0810 22:34:25.456404 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:25.456407 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:25.457496 (+ 1089us) server_negotiation.cc:407] Connection header received
0810 22:34:25.458424 (+ 928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:25.458427 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:25.458483 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:25.458550 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:25.459848 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:25.460399 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:25.462430 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:25.462663 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:25.462769 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:25.467423 (+ 4654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.66:57811: BlockingRecv error: recv got EOF from 127.16.233.66:57811 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260810 22:34:25.469223 27353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:25.456556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:45513 (local address 127.16.233.66:57811)
0810 22:34:25.457333 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:25.457378 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:25.458214 (+ 836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:25.458721 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:25.458728 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:25.459137 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:25.459633 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:25.459646 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:25.460559 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:25.460564 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:25.462254 (+ 1690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:25.462273 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:25.462847 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:25.462854 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:25.463121 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:25.463938 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:25.463957 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:25.467901 (+ 3944us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.126:45513: Server oryx/127.16.233.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":109,"threads_started":1}
W20260810 22:34:25.470062 27351 heartbeater.cc:644] Failed to heartbeat to 127.16.233.126:45513 (0 consecutive failures): Not authorized: Failed to ping master at 127.16.233.126:45513: Client connection negotiation failed: client connection to 127.16.233.126:45513: Server oryx/127.16.233.126@KRBTEST.COM not found in Kerberos database
Aug 10 22:34:25 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.66@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:34:25 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.66@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:34:25.476161 27353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:25.470597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:45513 (local address 127.16.233.66:36365)
0810 22:34:25.470722 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:25.470737 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:25.470852 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:25.471104 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:25.471107 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:25.471296 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:25.471397 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:25.471403 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:25.472160 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:25.472164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:25.472742 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:25.472750 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:25.473233 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:25.473237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:25.473339 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:25.473917 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:25.473928 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:25.476001 (+ 2073us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.126:45513: Server oryx/127.16.233.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":45}
W20260810 22:34:25.476269 27196 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:34:25.470718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:36365 (local address 127.16.233.126:45513)
0810 22:34:25.470865 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:25.470870 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:25.470888 (+ 18us) server_negotiation.cc:407] Connection header received
0810 22:34:25.470960 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:25.470963 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:25.471003 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:25.471085 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:25.471517 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:25.472050 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:25.472893 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:25.473120 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:25.473230 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:25.476146 (+ 2916us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.66:36365: BlockingRecv error: recv got EOF from 127.16.233.66:36365 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 10 22:34:25 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.66@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:34:25 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.66@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:34:25.482810 27353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:25.476851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:45513 (local address 127.16.233.66:37253)
0810 22:34:25.477060 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:25.477073 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:25.477146 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:25.477392 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:25.477394 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:25.477521 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:25.477594 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:25.477598 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:25.478641 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:25.478644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:25.479184 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:25.479199 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:25.479800 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:25.479803 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:25.479867 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:25.480434 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:25.480447 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:25.482686 (+ 2239us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.126:45513: Server oryx/127.16.233.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":47}
W20260810 22:34:25.482976 27196 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:34:25.476898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:37253 (local address 127.16.233.126:45513)
0810 22:34:25.477016 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:25.477020 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:25.477131 (+ 111us) server_negotiation.cc:407] Connection header received
0810 22:34:25.477228 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:25.477231 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:25.477278 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:25.477376 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:25.477694 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:25.478506 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:25.479300 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:25.479661 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:25.479764 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:25.482853 (+ 3089us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.66:37253: BlockingRecv error: recv got EOF from 127.16.233.66:37253 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260810 22:34:25.483115 27351 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 22:34:25.585340 27354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:34:25.585657 27354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:34:25.585705 27354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:34:25.589977 27354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:34:25.590052 27354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:34:25.590116 27354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.67
I20260810 22:34:25.595938 27354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:37427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/krb5kdc/oryx_127.16.233.67.keytab
--principal=oryx/127.16.233.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.67:38497
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.233.67
--webserver_port=39573
--tserver_master_addrs=127.16.233.126:45513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.67
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:34:25.597301 27354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:34:25.598415 27354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:34:25.609333 27361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:25.609457 27354 server_base.cc:1034] running on GCE node
W20260810 22:34:25.609335 27364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:25.609333 27362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:25.610144 27354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:34:25.610904 27354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:34:25.612136 27354 hybrid_clock.cc:648] HybridClock initialized: now 1786401265612109 us; error 42 us; skew 500 ppm
Aug 10 22:34:25 dist-test-slave-77v9 krb5kdc[26343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401265, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.233.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:34:25.615444 27354 init.cc:377] Logged in from keytab as oryx/127.16.233.67@KRBTEST.COM (short username oryx)
I20260810 22:34:25.616480 27354 webserver.cc:460] Webserver started at http://127.16.233.67:39573/ using document root <none> and password file <none>
I20260810 22:34:25.617165 27354 fs_manager.cc:359] Metadata directory not provided
I20260810 22:34:25.617293 27354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:34:25.621583 27354 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260810 22:34:25.623886 27371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:25.625367 27354 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 22:34:25.625520 27354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-2/wal
uuid: "c33b3eff30b14037b3c6552d562a89c5"
format_stamp: "Formatted at 2026-08-10 22:34:24 on dist-test-slave-77v9"
server_key: "c4277767bb40cb6752a7f6446c2fc430"
server_key_iv: "be28400aca87e98b914830098bdb54ba"
server_key_version: "encryptionkey@0"
I20260810 22:34:25.626165 27354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:34:25.647097 27354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:34:25.650724 27354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:34:25.650949 27354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:34:25.651598 27354 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:34:25.652480 27354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:34:25.652550 27354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:25.652619 27354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:34:25.652666 27354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:25.663406 27354 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.67:38497
I20260810 22:34:25.663528 27501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.67:38497 every 8 connection(s)
I20260810 22:34:25.664605 27354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
I20260810 22:34:25.670785 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 27354
Aug 10 22:34:25 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.67@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:34:25 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.67@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
W20260810 22:34:25.677760 27196 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:34:25.666923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:56991 (local address 127.16.233.126:45513)
0810 22:34:25.667092 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:25.667098 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:25.668371 (+ 1273us) server_negotiation.cc:407] Connection header received
0810 22:34:25.669471 (+ 1100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:25.669474 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:25.669527 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:25.669595 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:25.670954 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:25.671708 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:25.673251 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:25.673495 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:25.673614 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:25.677641 (+ 4027us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.67:56991: BlockingRecv error: recv got EOF from 127.16.233.67:56991 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 22:34:25.678905 27504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:25.667279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:45513 (local address 127.16.233.67:56991)
0810 22:34:25.668186 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:25.668232 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:25.669236 (+ 1004us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:25.669790 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:25.669797 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:25.670202 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:25.670778 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:25.670793 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:25.671901 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:25.671904 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:25.673114 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:25.673125 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:25.673661 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:25.673665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:25.673849 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:25.674509 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:25.674528 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:25.677538 (+ 3010us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.126:45513: Server oryx/127.16.233.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":706,"thread_start_us":125,"threads_started":1}
W20260810 22:34:25.679430 27502 heartbeater.cc:644] Failed to heartbeat to 127.16.233.126:45513 (0 consecutive failures): Not authorized: Failed to ping master at 127.16.233.126:45513: Client connection negotiation failed: client connection to 127.16.233.126:45513: Server oryx/127.16.233.126@KRBTEST.COM not found in Kerberos database
I20260810 22:34:25.682631 27506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:25.671930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42580 (local address 127.16.233.126:45513)
0810 22:34:25.672233 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:25.672238 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:25.672252 (+ 14us) server_negotiation.cc:407] Connection header received
0810 22:34:25.672291 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:25.672294 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:25.672340 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:25.672445 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:25.673340 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:25.673945 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:25.674813 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:25.675041 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:25.675145 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:25.676837 (+ 1692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:25.676849 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:25.676871 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:25.676909 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:25.679860 (+ 2951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:25.680556 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:25.680563 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:25.680569 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:25.680752 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:25.681171 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:25.681174 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:25.681175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:25.681569 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:25.681730 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:25.682164 (+ 434us) server_negotiation.cc:299] Negotiation successful
0810 22:34:25.682459 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":97,"threads_started":1}
Aug 10 22:34:25 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.67@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:34:25 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.67@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:34:25.686080 27504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:25.679983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:45513 (local address 127.16.233.67:51447)
0810 22:34:25.680179 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:25.680192 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:25.680258 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:25.680544 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:25.680547 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:25.680736 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:25.680987 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:25.680993 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:25.682004 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:25.682008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:25.682811 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:25.682828 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:25.683288 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:25.683291 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:25.683392 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:25.683872 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:25.683899 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:25.685933 (+ 2034us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.126:45513: Server oryx/127.16.233.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115}
W20260810 22:34:25.686144 27196 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:34:25.680058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:51447 (local address 127.16.233.126:45513)
0810 22:34:25.680199 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:25.680204 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:25.680221 (+ 17us) server_negotiation.cc:407] Connection header received
0810 22:34:25.680336 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:25.680340 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:25.680390 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:25.680482 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:25.681113 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:25.681790 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:25.682950 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:25.683170 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:25.683274 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:25.686038 (+ 2764us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.67:51447: BlockingRecv error: recv got EOF from 127.16.233.67:51447 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 10 22:34:25 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.67@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:34:25 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.67@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:34:25.691841 27504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:25.686736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:45513 (local address 127.16.233.67:43587)
0810 22:34:25.686855 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:25.686869 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:25.686942 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:25.687226 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:25.687228 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:25.687353 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:25.687434 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:25.687438 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:25.688246 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:25.688248 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:25.688759 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:25.688767 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:25.689382 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:25.689386 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:25.689448 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:25.689919 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:25.689931 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:25.691738 (+ 1807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.126:45513: Server oryx/127.16.233.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":48}
W20260810 22:34:25.692066 27196 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:34:25.686886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:43587 (local address 127.16.233.126:45513)
0810 22:34:25.687036 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:25.687040 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:25.687052 (+ 12us) server_negotiation.cc:407] Connection header received
0810 22:34:25.687085 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:25.687087 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:25.687124 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:25.687196 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:25.687559 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:25.688119 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:25.688929 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:25.689244 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:25.689343 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:25.691907 (+ 2564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.67:43587: BlockingRecv error: recv got EOF from 127.16.233.67:43587 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260810 22:34:25.692113 27502 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 10 22:34:26 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.65@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:34:26 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.65@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:34:26.286675 27507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:26.280276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:45513 (local address 127.16.233.65:50447)
0810 22:34:26.280621 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:26.280636 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:26.280738 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:26.281067 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:26.281070 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:26.281262 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:26.281396 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:26.281402 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:26.282257 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:26.282275 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:26.282975 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:26.282987 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:26.283499 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:26.283503 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:26.283603 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:26.284358 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:26.284371 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:26.286547 (+ 2176us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.126:45513: Server oryx/127.16.233.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":109,"threads_started":1}
W20260810 22:34:26.287056 27508 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:34:26.280406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:50447 (local address 127.16.233.126:45513)
0810 22:34:26.280699 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:26.280703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:26.280720 (+ 17us) server_negotiation.cc:407] Connection header received
0810 22:34:26.280882 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:26.280885 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:26.280935 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:26.281022 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:26.281529 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:26.282101 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:26.283111 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:26.283370 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:26.283490 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:26.286897 (+ 3407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.65:50447: BlockingRecv error: recv got EOF from 127.16.233.65:50447 (error 108)
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":84,"threads_started":1}
Aug 10 22:34:26 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.66@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:34:26 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.66@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:34:26.490222 27510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:26.483942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:45513 (local address 127.16.233.66:37191)
0810 22:34:26.484280 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:26.484297 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:26.484428 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:26.484695 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:26.484697 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:26.484965 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:26.485067 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:26.485073 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:26.485858 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:26.485861 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:26.486428 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:26.486435 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:26.487109 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:26.487120 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:26.487192 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:26.487840 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:26.487851 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:26.490036 (+ 2185us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.126:45513: Server oryx/127.16.233.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":94,"threads_started":1}
W20260810 22:34:26.490290 27508 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:34:26.484059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:37191 (local address 127.16.233.126:45513)
0810 22:34:26.484272 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:26.484277 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:26.484391 (+ 114us) server_negotiation.cc:407] Connection header received
0810 22:34:26.484530 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:26.484533 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:26.484580 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:26.484656 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:26.485186 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:26.485728 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:26.486637 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:26.486964 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:26.487075 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:26.490175 (+ 3100us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.66:37191: BlockingRecv error: recv got EOF from 127.16.233.66:37191 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
Aug 10 22:34:26 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.67@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:34:26 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.67@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:34:26.699820 27511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:26.692925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:45513 (local address 127.16.233.67:48859)
0810 22:34:26.693277 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:26.693297 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:26.693461 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:26.693760 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:26.693763 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:26.694053 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:26.694158 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:26.694165 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:26.695039 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:26.695045 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:26.695870 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:26.695881 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:26.696697 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:26.696703 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:26.696883 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:26.697545 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:26.697559 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:26.699678 (+ 2119us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.126:45513: Server oryx/127.16.233.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":98,"threads_started":1}
W20260810 22:34:26.700033 27508 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:34:26.693084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:48859 (local address 127.16.233.126:45513)
0810 22:34:26.693262 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:26.693277 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:26.693388 (+ 111us) server_negotiation.cc:407] Connection header received
0810 22:34:26.693574 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:26.693578 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:26.693637 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:26.693736 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:26.694325 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:26.694916 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:26.696031 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:26.696448 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:26.696608 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:26.699843 (+ 3235us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.67:48859: BlockingRecv error: recv got EOF from 127.16.233.67:48859 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 10 22:34:27 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.65@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:34:27 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.65@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:34:27.294028 27512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:27.287649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:45513 (local address 127.16.233.65:59051)
0810 22:34:27.288121 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:27.288144 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:27.288308 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:27.288606 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:27.288608 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:27.288804 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:27.289020 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:27.289027 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:27.289945 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:27.289948 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:27.290540 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:27.290549 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:27.291108 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:27.291111 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:27.291182 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:27.291811 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:27.291824 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:27.293840 (+ 2016us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.126:45513: Server oryx/127.16.233.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":107,"threads_started":1}
W20260810 22:34:27.294315 27514 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:34:27.287814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:59051 (local address 127.16.233.126:45513)
0810 22:34:27.288223 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:27.288228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:27.288245 (+ 17us) server_negotiation.cc:407] Connection header received
0810 22:34:27.288406 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:27.288408 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:27.288458 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:27.288547 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:27.289169 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:27.289760 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:27.290681 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:27.290984 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:27.291082 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:27.294167 (+ 3085us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.65:59051: BlockingRecv error: recv got EOF from 127.16.233.65:59051 (error 108)
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":164,"threads_started":1}
Aug 10 22:34:27 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.66@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:34:27 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.66@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:34:27.497726 27515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:27.491181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:45513 (local address 127.16.233.66:60255)
0810 22:34:27.491583 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:27.491601 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:27.491754 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:27.492183 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:27.492185 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:27.492392 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:27.492494 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:27.492520 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:27.493639 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:27.493642 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:27.494281 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:27.494291 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:27.494818 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:27.494822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:27.494892 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:27.495579 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:27.495591 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:27.497566 (+ 1975us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.126:45513: Server oryx/127.16.233.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":132,"threads_started":1}
W20260810 22:34:27.497905 27514 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:34:27.491357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:60255 (local address 127.16.233.126:45513)
0810 22:34:27.491563 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:27.491568 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:27.491692 (+ 124us) server_negotiation.cc:407] Connection header received
0810 22:34:27.491954 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:27.491958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:27.492013 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:27.492139 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:27.492642 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:27.493479 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:27.494404 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:27.494651 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:27.494812 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:27.497737 (+ 2925us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.66:60255: BlockingRecv error: recv got EOF from 127.16.233.66:60255 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Aug 10 22:34:27 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.67@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:34:27 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.67@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:34:27.707234 27516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:27.700933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:45513 (local address 127.16.233.67:51733)
0810 22:34:27.701245 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:27.701260 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:27.701373 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:27.701627 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:27.701629 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:27.701817 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:27.701918 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:27.701924 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:27.702746 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:27.702748 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:27.703441 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:27.703454 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:27.704100 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:27.704106 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:27.704214 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:27.704882 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:27.704899 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:27.707058 (+ 2159us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.126:45513: Server oryx/127.16.233.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":79,"threads_started":1}
W20260810 22:34:27.707329 27514 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:34:27.701043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:51733 (local address 127.16.233.126:45513)
0810 22:34:27.701238 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:27.701242 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:27.701333 (+ 91us) server_negotiation.cc:407] Connection header received
0810 22:34:27.701460 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:27.701463 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:27.701511 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:27.701586 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:27.702061 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:27.702629 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:27.703605 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:27.703936 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:27.704067 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:27.707218 (+ 3151us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.67:51733: BlockingRecv error: recv got EOF from 127.16.233.67:51733 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Aug 10 22:34:28 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.65@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:34:28 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.65@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:34:28.301043 27517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:28.294911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:45513 (local address 127.16.233.65:37381)
0810 22:34:28.295207 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:28.295246 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:28.295365 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:28.295625 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:28.295627 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:28.295825 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:28.295930 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:28.295935 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:28.297035 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:28.297039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:28.297700 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:28.297709 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:28.298271 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:28.298275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:28.298355 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:28.298930 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:28.298942 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:28.300899 (+ 1957us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.126:45513: Server oryx/127.16.233.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":87,"threads_started":1}
W20260810 22:34:28.301334 27518 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:34:28.295101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:37381 (local address 127.16.233.126:45513)
0810 22:34:28.295417 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:28.295421 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:28.295435 (+ 14us) server_negotiation.cc:407] Connection header received
0810 22:34:28.295471 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:28.295473 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:28.295515 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:28.295608 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:28.296064 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:28.296909 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:28.297896 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:28.298138 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:28.298232 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:28.301156 (+ 2924us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.65:37381: BlockingRecv error: recv got EOF from 127.16.233.65:37381 (error 108)
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":94,"threads_started":1}
Aug 10 22:34:28 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.66@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:34:28 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.66@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:34:28.504931 27519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:28.498650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:45513 (local address 127.16.233.66:51361)
0810 22:34:28.498968 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:28.498983 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:28.499091 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:28.499372 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:28.499375 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:28.499567 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:28.499676 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:28.499681 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:28.500542 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:28.500545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:28.501216 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:28.501229 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:28.501770 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:28.501775 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:28.501853 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:28.502504 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:28.502527 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:28.504705 (+ 2178us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.126:45513: Server oryx/127.16.233.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":131,"threads_started":1}
W20260810 22:34:28.504997 27518 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:34:28.498744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:51361 (local address 127.16.233.126:45513)
0810 22:34:28.498912 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:28.498916 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:28.499060 (+ 144us) server_negotiation.cc:407] Connection header received
0810 22:34:28.499186 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:28.499189 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:28.499240 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:28.499324 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:28.499815 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:28.500415 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:28.501366 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:28.501601 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:28.501753 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:28.504883 (+ 3130us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.66:51361: BlockingRecv error: recv got EOF from 127.16.233.66:51361 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 10 22:34:28 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.67@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:34:28 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.67@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:34:28.714555 27520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:28.708337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:45513 (local address 127.16.233.67:51043)
0810 22:34:28.708708 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:28.708730 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:28.708878 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:28.709156 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:28.709159 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:28.709455 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:28.709569 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:28.709575 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:28.710536 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:28.710539 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:28.711116 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:28.711124 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:28.711605 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:28.711607 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:28.711669 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:28.712246 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:28.712265 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:28.714428 (+ 2163us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.126:45513: Server oryx/127.16.233.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":131,"threads_started":1}
W20260810 22:34:28.714713 27518 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:34:28.708387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:51043 (local address 127.16.233.126:45513)
0810 22:34:28.708573 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:28.708577 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:28.708792 (+ 215us) server_negotiation.cc:407] Connection header received
0810 22:34:28.708984 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:28.708987 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:28.709043 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:28.709129 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:28.709776 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:28.710405 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:28.711243 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:28.711490 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:28.711579 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:28.714584 (+ 3005us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.67:51043: BlockingRecv error: recv got EOF from 127.16.233.67:51043 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Aug 10 22:34:29 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.65@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:34:29 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.65@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:34:29.314360 27521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:29.302051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:45513 (local address 127.16.233.65:55789)
0810 22:34:29.302604 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:29.302618 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:29.302724 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:29.303021 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:29.303023 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:29.303218 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:29.303319 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:29.303324 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:29.304234 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:29.304237 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:29.304947 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:29.304957 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:29.305421 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:29.305424 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:29.305485 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:29.312125 (+ 6640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:29.312139 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:29.314235 (+ 2096us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.126:45513: Server oryx/127.16.233.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":356,"threads_started":1}
W20260810 22:34:29.314618 27523 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:34:29.302117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:55789 (local address 127.16.233.126:45513)
0810 22:34:29.302605 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:29.302609 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:29.302672 (+ 63us) server_negotiation.cc:407] Connection header received
0810 22:34:29.302825 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:29.302829 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:29.302883 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:29.302967 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:29.303437 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:29.304118 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:29.305055 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:29.305303 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:29.305387 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:29.314444 (+ 9057us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.65:55789: BlockingRecv error: recv got EOF from 127.16.233.65:55789 (error 108)
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":291,"threads_started":1}
Aug 10 22:34:29 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.66@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:34:29 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.66@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:34:29.511646 27526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:29.505815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:45513 (local address 127.16.233.66:53587)
0810 22:34:29.506106 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:29.506121 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:29.506224 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:29.506491 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:29.506493 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:29.506685 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:29.506784 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:29.506789 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:29.507588 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:29.507590 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:29.508146 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:29.508154 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:29.508634 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:29.508636 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:29.508699 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:29.509313 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:29.509326 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:29.511514 (+ 2188us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.126:45513: Server oryx/127.16.233.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":104,"threads_started":1}
W20260810 22:34:29.511806 27523 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:34:29.505868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:53587 (local address 127.16.233.126:45513)
0810 22:34:29.506044 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:29.506048 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:29.506221 (+ 173us) server_negotiation.cc:407] Connection header received
0810 22:34:29.506315 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:29.506317 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:29.506370 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:29.506461 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:29.506923 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:29.507481 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:29.508280 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:29.508515 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:29.508615 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:29.511672 (+ 3057us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.66:53587: BlockingRecv error: recv got EOF from 127.16.233.66:53587 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 10 22:34:29 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.67@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:34:29 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.67@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:34:29.721779 27527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:29.715410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:45513 (local address 127.16.233.67:42083)
0810 22:34:29.715706 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:29.715720 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:29.715890 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:29.716184 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:29.716186 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:29.716395 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:29.716495 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:29.716500 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:29.717432 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:29.717435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:29.718005 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:29.718014 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:29.718519 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:29.718522 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:29.718595 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:29.719227 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:29.719239 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:29.721585 (+ 2346us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.126:45513: Server oryx/127.16.233.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":120,"threads_started":1}
W20260810 22:34:29.722003 27523 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:34:29.715539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:42083 (local address 127.16.233.126:45513)
0810 22:34:29.715681 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:29.715685 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:29.715832 (+ 147us) server_negotiation.cc:407] Connection header received
0810 22:34:29.716001 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:29.716004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:29.716060 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:29.716146 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:29.716611 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:29.717285 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:29.718135 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:29.718390 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:29.718480 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:29.721872 (+ 3392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.67:42083: BlockingRecv error: recv got EOF from 127.16.233.67:42083 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 10 22:34:30 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.65@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:34:30 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.65@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:34:30.321699 27528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:30.315467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:45513 (local address 127.16.233.65:59659)
0810 22:34:30.315780 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:30.315795 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:30.315885 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:30.316187 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:30.316190 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:30.316404 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:30.316515 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:30.316521 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:30.317409 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:30.317413 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:30.318034 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:30.318045 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:30.318575 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:30.318579 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:30.318662 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:30.319335 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:30.319347 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:30.321560 (+ 2213us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.126:45513: Server oryx/127.16.233.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":105,"threads_started":1}
W20260810 22:34:30.321967 27529 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:34:30.315544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:59659 (local address 127.16.233.126:45513)
0810 22:34:30.315861 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:30.315864 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:30.315879 (+ 15us) server_negotiation.cc:407] Connection header received
0810 22:34:30.315973 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:30.315975 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:30.316020 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:30.316100 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:30.316666 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:30.317264 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:30.318173 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:30.318440 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:30.318533 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:30.321775 (+ 3242us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.65:59659: BlockingRecv error: recv got EOF from 127.16.233.65:59659 (error 108)
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":163,"threads_started":1}
Aug 10 22:34:30 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.66@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:34:30 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.66@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:34:30.518823 27530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:30.512651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:45513 (local address 127.16.233.66:47691)
0810 22:34:30.513040 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:30.513059 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:30.513194 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:30.513491 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:30.513494 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:30.513729 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:30.513864 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:30.513870 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:30.514663 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:30.514668 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:30.515242 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:30.515250 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:30.515709 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:30.515713 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:30.515810 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:30.516412 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:30.516425 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:30.518633 (+ 2208us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.126:45513: Server oryx/127.16.233.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":177,"threads_started":1}
W20260810 22:34:30.519004 27529 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:34:30.512885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:47691 (local address 127.16.233.126:45513)
0810 22:34:30.513069 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:30.513074 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:30.513132 (+ 58us) server_negotiation.cc:407] Connection header received
0810 22:34:30.513311 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:30.513314 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:30.513372 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:30.513466 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:30.514011 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:30.514552 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:30.515365 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:30.515592 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:30.515701 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:30.518886 (+ 3185us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.66:47691: BlockingRecv error: recv got EOF from 127.16.233.66:47691 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 10 22:34:30 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.67@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:34:30 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.67@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:34:30.729709 27531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:30.722882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:45513 (local address 127.16.233.67:39547)
0810 22:34:30.723173 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:30.723189 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:30.723289 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:30.723647 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:30.723650 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:30.723966 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:30.724082 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:30.724090 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:30.725095 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:30.725098 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:30.725821 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:30.725837 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:30.726359 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:30.726362 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:30.726473 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:30.727179 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:30.727193 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:30.729573 (+ 2380us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.126:45513: Server oryx/127.16.233.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":85,"threads_started":1}
W20260810 22:34:30.729941 27529 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:34:30.723046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:39547 (local address 127.16.233.126:45513)
0810 22:34:30.723244 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:30.723248 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:30.723282 (+ 34us) server_negotiation.cc:407] Connection header received
0810 22:34:30.723423 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:30.723426 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:30.723482 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:30.723582 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:30.724227 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:30.724972 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:30.725971 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:30.726250 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:30.726340 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:30.729763 (+ 3423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.67:39547: BlockingRecv error: recv got EOF from 127.16.233.67:39547 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
Aug 10 22:34:31 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.65@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:34:31 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.65@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:34:31.329468 27532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:31.322824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:45513 (local address 127.16.233.65:36789)
0810 22:34:31.323128 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:31.323142 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:31.323233 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:31.323508 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:31.323511 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:31.323699 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:31.323881 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:31.323888 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:31.324723 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:31.324727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:31.325575 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:31.325586 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:31.326142 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:31.326145 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:31.326338 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:31.327000 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:31.327014 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:31.329329 (+ 2315us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.126:45513: Server oryx/127.16.233.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":122,"threads_started":1}
W20260810 22:34:31.329831 27533 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:34:31.322927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:36789 (local address 127.16.233.126:45513)
0810 22:34:31.323192 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:31.323196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:31.323211 (+ 15us) server_negotiation.cc:407] Connection header received
0810 22:34:31.323342 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:31.323344 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:31.323389 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:31.323474 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:31.324009 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:31.324589 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:31.325712 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:31.325992 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:31.326094 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:31.329541 (+ 3447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.65:36789: BlockingRecv error: recv got EOF from 127.16.233.65:36789 (error 108)
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":101,"threads_started":1}
Aug 10 22:34:31 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.66@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:34:31 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.66@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:34:31.525309 27534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:31.519776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:45513 (local address 127.16.233.66:42193)
0810 22:34:31.520079 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:31.520093 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:31.520195 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:31.520442 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:31.520444 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:31.520637 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:31.520733 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:31.520739 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:31.521562 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:31.521565 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:31.522127 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:31.522135 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:31.522588 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:31.522590 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:31.522653 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:31.523244 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:31.523256 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:31.525177 (+ 1921us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.126:45513: Server oryx/127.16.233.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":118,"threads_started":1}
W20260810 22:34:31.525588 27533 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:34:31.519859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:42193 (local address 127.16.233.126:45513)
0810 22:34:31.520027 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:31.520035 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:31.520165 (+ 130us) server_negotiation.cc:407] Connection header received
0810 22:34:31.520275 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:31.520278 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:31.520324 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:31.520407 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:31.520911 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:31.521455 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:31.522248 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:31.522482 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:31.522565 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:31.525454 (+ 2889us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.66:42193: BlockingRecv error: recv got EOF from 127.16.233.66:42193 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Aug 10 22:34:31 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.67@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:34:31 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.67@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:34:31.737109 27536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:31.730886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:45513 (local address 127.16.233.67:46685)
0810 22:34:31.731211 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:31.731229 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:31.731381 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:31.731660 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:31.731663 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:31.731904 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:31.732000 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:31.732006 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:31.732777 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:31.732781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:31.733372 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:31.733381 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:31.733849 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:31.733853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:31.733932 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:31.734496 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:31.734509 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:31.736969 (+ 2460us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.126:45513: Server oryx/127.16.233.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":90,"threads_started":1}
W20260810 22:34:31.737290 27533 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:34:31.730999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:46685 (local address 127.16.233.126:45513)
0810 22:34:31.731170 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:31.731200 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:31.731319 (+ 119us) server_negotiation.cc:407] Connection header received
0810 22:34:31.731484 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:31.731490 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:31.731547 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:31.731639 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:31.732128 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:31.732666 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:31.733495 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:31.733733 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:31.733842 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:31.737141 (+ 3299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.67:46685: BlockingRecv error: recv got EOF from 127.16.233.67:46685 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 10 22:34:32 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.65@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:34:32 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.65@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:34:32.336762 27537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:32.330495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:45513 (local address 127.16.233.65:42567)
0810 22:34:32.330865 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:32.330882 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:32.330984 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:32.331315 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:32.331318 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:32.331540 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:32.331649 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:32.331654 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:32.332568 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:32.332571 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:32.333284 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:32.333296 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:32.333952 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:32.333955 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:32.334030 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:32.334627 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:32.334639 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:32.336631 (+ 1992us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.126:45513: Server oryx/127.16.233.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":119,"threads_started":1}
W20260810 22:34:32.337144 27538 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:34:32.330575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:42567 (local address 127.16.233.126:45513)
0810 22:34:32.330914 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:32.330918 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:32.330936 (+ 18us) server_negotiation.cc:407] Connection header received
0810 22:34:32.331131 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:32.331134 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:32.331182 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:32.331272 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:32.331837 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:32.332420 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:32.333458 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:32.333721 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:32.333906 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:32.336978 (+ 3072us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.65:42567: BlockingRecv error: recv got EOF from 127.16.233.65:42567 (error 108)
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":112,"threads_started":1}
Aug 10 22:34:32 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.66@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:34:32 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.66@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:34:32.533115 27539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:32.526306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:45513 (local address 127.16.233.66:53977)
0810 22:34:32.526680 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:32.526696 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:32.526894 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:32.527161 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:32.527163 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:32.527404 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:32.527505 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:32.527512 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:32.528407 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:32.528410 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:32.529094 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:32.529105 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:32.529761 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:32.529764 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:32.529905 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:32.530594 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:32.530625 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:32.532968 (+ 2343us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.126:45513: Server oryx/127.16.233.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":122,"threads_started":1}
W20260810 22:34:32.533344 27538 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:34:32.526399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:53977 (local address 127.16.233.126:45513)
0810 22:34:32.526587 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:32.526593 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:32.526891 (+ 298us) server_negotiation.cc:407] Connection header received
0810 22:34:32.526995 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:32.526997 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:32.527048 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:32.527117 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:32.527628 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:32.528278 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:32.529226 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:32.529597 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:32.529728 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:32.533194 (+ 3466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.66:53977: BlockingRecv error: recv got EOF from 127.16.233.66:53977 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 10 22:34:32 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.67@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:34:32 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.67@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:34:32.744405 27540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:32.737994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:45513 (local address 127.16.233.67:46687)
0810 22:34:32.738325 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:32.738339 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:32.738463 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:32.738847 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:32.738850 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:32.739060 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:32.739174 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:32.739181 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:32.739985 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:32.739988 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:32.740565 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:32.740575 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:32.741141 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:32.741145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:32.741228 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:32.742032 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:32.742047 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:32.744253 (+ 2206us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.126:45513: Server oryx/127.16.233.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":80,"threads_started":1}
W20260810 22:34:32.744781 27538 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:34:32.738129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:46687 (local address 127.16.233.126:45513)
0810 22:34:32.738328 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:32.738356 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:32.738417 (+ 61us) server_negotiation.cc:407] Connection header received
0810 22:34:32.738551 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:32.738553 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:32.738603 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:32.738688 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:32.739303 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:32.739862 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:32.740688 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:32.740995 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:32.741090 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:32.744625 (+ 3535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.67:46687: BlockingRecv error: recv got EOF from 127.16.233.67:46687 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Aug 10 22:34:33 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.65@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:34:33 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.65@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:34:33.344172 27541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:33.337931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:45513 (local address 127.16.233.65:52173)
0810 22:34:33.338255 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:33.338270 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:33.338368 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:33.338725 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:33.338727 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:33.338932 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:33.339036 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:33.339042 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:33.339913 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:33.339915 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:33.340488 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:33.340496 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:33.341035 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:33.341043 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:33.341121 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:33.341787 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:33.341801 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:33.344043 (+ 2242us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.126:45513: Server oryx/127.16.233.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":119,"threads_started":1}
W20260810 22:34:33.344331 27542 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:34:33.338184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:52173 (local address 127.16.233.126:45513)
0810 22:34:33.338449 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:33.338453 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:33.338469 (+ 16us) server_negotiation.cc:407] Connection header received
0810 22:34:33.338514 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:33.338516 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:33.338573 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:33.338672 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:33.339172 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:33.339793 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:33.340618 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:33.340896 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:33.340987 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:33.344194 (+ 3207us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.65:52173: BlockingRecv error: recv got EOF from 127.16.233.65:52173 (error 108)
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":98,"threads_started":1}
Aug 10 22:34:33 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.66@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:34:33 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.66@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:34:33.541006 27544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:33.534298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:45513 (local address 127.16.233.66:49735)
0810 22:34:33.534630 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:33.534646 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:33.534773 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:33.535219 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:33.535222 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:33.535422 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:33.535534 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:33.535540 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:33.536519 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:33.536521 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:33.537179 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:33.537189 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:33.537651 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:33.537653 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:33.537718 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:33.538323 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:33.538335 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:33.540798 (+ 2463us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.126:45513: Server oryx/127.16.233.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":85,"threads_started":1}
W20260810 22:34:33.541091 27542 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:34:33.534389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:49735 (local address 127.16.233.126:45513)
0810 22:34:33.534616 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:33.534620 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:33.534759 (+ 139us) server_negotiation.cc:407] Connection header received
0810 22:34:33.534908 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:33.534911 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:33.534967 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:33.535064 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:33.535666 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:33.536406 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:33.537298 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:33.537531 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:33.537623 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:33.540970 (+ 3347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.66:49735: BlockingRecv error: recv got EOF from 127.16.233.66:49735 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Aug 10 22:34:33 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.67@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:34:33 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.67@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:34:33.752202 27545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:33.745599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:45513 (local address 127.16.233.67:53397)
0810 22:34:33.745925 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:33.745939 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:33.746045 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:33.746283 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:33.746286 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:33.746486 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:33.746585 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:33.746590 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:33.747529 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:33.747532 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:33.748183 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:33.748194 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:33.748716 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:33.748719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:33.748784 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:33.749545 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:33.749602 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:33.752081 (+ 2479us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.126:45513: Server oryx/127.16.233.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":92,"threads_started":1}
W20260810 22:34:33.752524 27542 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:34:33.745875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:53397 (local address 127.16.233.126:45513)
0810 22:34:33.746041 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:33.746045 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:33.746062 (+ 17us) server_negotiation.cc:407] Connection header received
0810 22:34:33.746110 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:33.746113 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:33.746165 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:33.746249 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:33.746732 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:33.747373 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:33.748348 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:33.748594 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:33.748689 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:33.752388 (+ 3699us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.67:53397: BlockingRecv error: recv got EOF from 127.16.233.67:53397 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Aug 10 22:34:34 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.65@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:34:34 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.65@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:34:34.356904 27547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:34.345176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:45513 (local address 127.16.233.65:41671)
0810 22:34:34.345493 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:34.345508 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:34.345635 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:34.346076 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:34.346081 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:34.346308 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:34.346423 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:34.346430 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:34.347361 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:34.347364 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:34.347961 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:34.347970 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:34.348487 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:34.348490 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:34.348552 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:34.354103 (+ 5551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:34.354120 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:34.356681 (+ 2561us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.126:45513: Server oryx/127.16.233.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":103,"threads_started":1}
W20260810 22:34:34.357131 27548 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:34:34.345265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:41671 (local address 127.16.233.126:45513)
0810 22:34:34.345527 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:34.345531 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:34.345600 (+ 69us) server_negotiation.cc:407] Connection header received
0810 22:34:34.345809 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:34.345811 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:34.345911 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:34.345993 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:34.346585 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:34.347240 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:34.348111 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:34.348377 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:34.348472 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:34.356936 (+ 8464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.65:41671: BlockingRecv error: recv got EOF from 127.16.233.65:41671 (error 108)
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":66,"threads_started":1}
Aug 10 22:34:34 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.66@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:34:34 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.66@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:34:34.548863 27549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:34.542177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:45513 (local address 127.16.233.66:38723)
0810 22:34:34.542535 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:34.542550 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:34.542722 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:34.543108 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:34.543111 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:34.543366 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:34.543480 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:34.543487 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:34.544366 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:34.544374 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:34.545053 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:34.545063 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:34.545663 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:34.545667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:34.545770 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:34.546430 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:34.546444 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:34.548631 (+ 2187us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.126:45513: Server oryx/127.16.233.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":108,"threads_started":1}
W20260810 22:34:34.548938 27548 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:34:34.542167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:38723 (local address 127.16.233.126:45513)
0810 22:34:34.542383 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:34.542390 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:34.542648 (+ 258us) server_negotiation.cc:407] Connection header received
0810 22:34:34.542914 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:34.542919 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:34.542980 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:34.543084 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:34.543627 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:34.544223 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:34.545206 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:34.545465 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:34.545651 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:34.548775 (+ 3124us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.66:38723: BlockingRecv error: recv got EOF from 127.16.233.66:38723 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Aug 10 22:34:34 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.67@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:34:34 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.67@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:34:34.760169 27553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:34.753205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:45513 (local address 127.16.233.67:53537)
0810 22:34:34.753699 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:34.753721 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:34.753946 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:34.754295 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:34.754298 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:34.754549 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:34.754650 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:34.754657 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:34.755524 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:34.755530 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:34.756168 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:34.756177 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:34.756741 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:34.756745 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:34.756894 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:34.757523 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:34.757537 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:34.759941 (+ 2404us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.126:45513: Server oryx/127.16.233.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":143,"threads_started":1}
W20260810 22:34:34.760247 27548 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:34:34.753432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:53537 (local address 127.16.233.126:45513)
0810 22:34:34.753658 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:34.753663 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:34.753805 (+ 142us) server_negotiation.cc:407] Connection header received
0810 22:34:34.754084 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:34.754088 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:34.754162 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:34.754264 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:34.754787 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:34.755396 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:34.756314 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:34.756603 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:34.756733 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:34.760114 (+ 3381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.67:53537: BlockingRecv error: recv got EOF from 127.16.233.67:53537 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
Aug 10 22:34:35 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.65@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:34:35 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.65@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:34:35.364163 27554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:35.358073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:45513 (local address 127.16.233.65:41991)
0810 22:34:35.358405 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:35.358419 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:35.358514 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:35.358848 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:35.358851 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:35.359090 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:35.359195 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:35.359201 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:35.360118 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:35.360121 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:35.360705 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:35.360713 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:35.361219 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:35.361222 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:35.361290 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:35.361885 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:35.361899 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:35.364021 (+ 2122us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.126:45513: Server oryx/127.16.233.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":131,"threads_started":1}
W20260810 22:34:35.364351 27555 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:34:35.358310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:41991 (local address 127.16.233.126:45513)
0810 22:34:35.358579 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:35.358584 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:35.358600 (+ 16us) server_negotiation.cc:407] Connection header received
0810 22:34:35.358648 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:35.358650 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:35.358694 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:35.358780 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:35.359316 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:35.360004 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:35.360863 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:35.361113 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:35.361194 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:35.364194 (+ 3000us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.65:41991: BlockingRecv error: recv got EOF from 127.16.233.65:41991 (error 108)
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":114,"threads_started":1}
Aug 10 22:34:35 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.66@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:34:35 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.66@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:34:35.556872 27556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:35.550006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:45513 (local address 127.16.233.66:49999)
0810 22:34:35.550332 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:35.550347 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:35.550467 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:35.550824 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:35.550827 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:35.551036 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:35.551144 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:35.551150 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:35.552213 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:35.552216 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:35.552967 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:35.552979 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:35.553507 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:35.553510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:35.553583 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:35.554159 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:35.554172 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:35.556633 (+ 2461us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.126:45513: Server oryx/127.16.233.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":116,"threads_started":1}
W20260810 22:34:35.556950 27555 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:34:35.550046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:49999 (local address 127.16.233.126:45513)
0810 22:34:35.550224 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:35.550231 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:35.550455 (+ 224us) server_negotiation.cc:407] Connection header received
0810 22:34:35.550545 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:35.550547 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:35.550606 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:35.550701 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:35.551335 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:35.552046 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:35.553107 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:35.553349 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:35.553444 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:35.556782 (+ 3338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.66:49999: BlockingRecv error: recv got EOF from 127.16.233.66:49999 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 10 22:34:35 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.67@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:34:35 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.67@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:34:35.768110 27557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:35.761184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:45513 (local address 127.16.233.67:48759)
0810 22:34:35.761659 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:35.761673 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:35.761783 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:35.762074 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:35.762076 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:35.762279 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:35.762419 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:35.762427 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:35.763323 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:35.763326 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:35.763911 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:35.763920 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:35.764591 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:35.764594 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:35.764664 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:35.765307 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:35.765319 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:35.767932 (+ 2613us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.126:45513: Server oryx/127.16.233.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":168,"threads_started":1}
W20260810 22:34:35.768299 27555 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:34:35.761482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:48759 (local address 127.16.233.126:45513)
0810 22:34:35.761650 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:35.761655 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:35.761750 (+ 95us) server_negotiation.cc:407] Connection header received
0810 22:34:35.761884 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:35.761886 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:35.761939 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:35.762024 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:35.762565 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:35.763183 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:35.764040 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:35.764324 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:35.764416 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:35.768133 (+ 3717us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.67:48759: BlockingRecv error: recv got EOF from 127.16.233.67:48759 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 10 22:34:36 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.65@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:34:36 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.65@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:34:36.371381 27558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:36.364998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:45513 (local address 127.16.233.65:39289)
0810 22:34:36.365339 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:36.365356 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:36.365481 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:36.365753 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:36.365755 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:36.365968 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:36.366074 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:36.366080 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:36.367017 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:36.367020 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:36.367710 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:36.367720 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:36.368350 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:36.368353 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:36.368430 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:36.369014 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:36.369027 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:36.371194 (+ 2167us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.126:45513: Server oryx/127.16.233.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":102,"threads_started":1}
W20260810 22:34:36.371518 27559 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:34:36.365073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:39289 (local address 127.16.233.126:45513)
0810 22:34:36.365357 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:36.365361 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:36.365441 (+ 80us) server_negotiation.cc:407] Connection header received
0810 22:34:36.365565 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:36.365567 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:36.365612 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:36.365691 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:36.366183 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:36.366812 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:36.367849 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:36.368235 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:36.368321 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:36.371349 (+ 3028us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.65:39289: BlockingRecv error: recv got EOF from 127.16.233.65:39289 (error 108)
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":88,"threads_started":1}
Aug 10 22:34:36 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.66@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:34:36 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.66@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:34:36.564477 27560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:36.557987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:45513 (local address 127.16.233.66:38965)
0810 22:34:36.558431 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:36.558448 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:36.558619 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:36.558956 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:36.558959 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:36.559192 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:36.559285 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:36.559290 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:36.560101 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:36.560106 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:36.560779 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:36.560788 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:36.561418 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:36.561422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:36.561512 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:36.562246 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:36.562268 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:36.564310 (+ 2042us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.126:45513: Server oryx/127.16.233.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":174,"threads_started":1}
W20260810 22:34:36.564639 27559 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:34:36.558194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:38965 (local address 127.16.233.126:45513)
0810 22:34:36.558434 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:36.558442 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:36.558552 (+ 110us) server_negotiation.cc:407] Connection header received
0810 22:34:36.558747 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:36.558750 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:36.558797 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:36.558929 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:36.559427 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:36.559979 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:36.560981 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:36.561287 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:36.561414 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:36.564493 (+ 3079us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.66:38965: BlockingRecv error: recv got EOF from 127.16.233.66:38965 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Aug 10 22:34:36 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.67@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:34:36 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.67@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:34:36.780236 27561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:36.769403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:45513 (local address 127.16.233.67:43473)
0810 22:34:36.769924 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:36.769956 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:36.770248 (+ 292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:36.770778 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:36.770781 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:36.771167 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:36.771348 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:36.771357 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:36.772643 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:36.772650 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:36.773647 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:36.773659 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:36.774480 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:36.774486 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:36.774668 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:36.775685 (+ 1017us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:36.775702 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:36.779888 (+ 4186us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.126:45513: Server oryx/127.16.233.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":153,"threads_started":1}
W20260810 22:34:36.780421 27559 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:34:36.769554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:43473 (local address 127.16.233.126:45513)
0810 22:34:36.769924 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:36.769931 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:36.770127 (+ 196us) server_negotiation.cc:407] Connection header received
0810 22:34:36.770471 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:36.770490 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:36.770572 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:36.770748 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:36.771545 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:36.772431 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:36.773858 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:36.774232 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:36.774455 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:36.780141 (+ 5686us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.67:43473: BlockingRecv error: recv got EOF from 127.16.233.67:43473 (error 108)
Metrics: {"server-negotiator.queue_time_us":138}
Aug 10 22:34:37 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.65@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:34:37 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.65@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:34:37.382771 27562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:37.372739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:45513 (local address 127.16.233.65:33017)
0810 22:34:37.373427 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:37.373454 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:37.373650 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:37.374235 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:37.374238 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:37.374546 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:37.374716 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:37.374723 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:37.376065 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:37.376068 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:37.377108 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:37.377124 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:37.378008 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:37.378013 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:37.378149 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:37.379223 (+ 1074us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:37.379240 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:37.382587 (+ 3347us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.126:45513: Server oryx/127.16.233.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":190,"threads_started":1}
W20260810 22:34:37.383091 27563 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:34:37.372920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:33017 (local address 127.16.233.126:45513)
0810 22:34:37.373417 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:37.373423 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:37.373614 (+ 191us) server_negotiation.cc:407] Connection header received
0810 22:34:37.373871 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:37.373875 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:37.373986 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:37.374150 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:37.374960 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:37.375815 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:37.377301 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:37.377792 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:37.378005 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:37.382880 (+ 4875us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.65:33017: BlockingRecv error: recv got EOF from 127.16.233.65:33017 (error 108)
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":129,"threads_started":1}
Aug 10 22:34:37 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.66@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:34:37 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.66@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:34:37.575213 27564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:37.565577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:45513 (local address 127.16.233.66:49107)
0810 22:34:37.565993 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:37.566019 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:37.566201 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:37.566630 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:37.566633 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:37.567083 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:37.567263 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:37.567280 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:37.568708 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:37.568711 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:37.569871 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:37.569886 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:37.570583 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:37.570586 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:37.570679 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:37.571587 (+ 908us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:37.571623 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:37.574903 (+ 3280us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.126:45513: Server oryx/127.16.233.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":149,"threads_started":1}
W20260810 22:34:37.575489 27563 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:34:37.565734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:49107 (local address 127.16.233.126:45513)
0810 22:34:37.565966 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:37.565978 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:37.566163 (+ 185us) server_negotiation.cc:407] Connection header received
0810 22:34:37.566329 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:37.566340 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:37.566421 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:37.566560 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:37.567474 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:37.568520 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:37.570078 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:37.570427 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:37.570574 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:37.575297 (+ 4723us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.66:49107: BlockingRecv error: recv got EOF from 127.16.233.66:49107 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Aug 10 22:34:37 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.67@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:34:37 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.67@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:34:37.791548 27565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:37.781996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:45513 (local address 127.16.233.67:37073)
0810 22:34:37.782417 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:37.782433 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:37.782602 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:37.783022 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:37.783024 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:37.783356 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:37.783519 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:37.783526 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:37.784904 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:37.784912 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:37.785942 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:37.785962 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:37.786820 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:37.786824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:37.786932 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:37.787814 (+ 882us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:37.787840 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:37.791336 (+ 3496us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.126:45513: Server oryx/127.16.233.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":154,"threads_started":1}
W20260810 22:34:37.792037 27563 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:34:37.782181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:37073 (local address 127.16.233.126:45513)
0810 22:34:37.782416 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:37.782421 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:37.782587 (+ 166us) server_negotiation.cc:407] Connection header received
0810 22:34:37.782739 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:37.782741 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:37.782820 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:37.782968 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:37.783700 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:37.784534 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:37.786172 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:37.786550 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:37.786730 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:37.791683 (+ 4953us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.67:37073: BlockingRecv error: recv got EOF from 127.16.233.67:37073 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Aug 10 22:34:38 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.65@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:34:38 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.65@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:34:38.389941 27566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:38.383969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:45513 (local address 127.16.233.65:58861)
0810 22:34:38.384239 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:38.384253 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:38.384380 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:38.384674 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:38.384676 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:38.384928 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:38.385037 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:38.385044 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:38.385946 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:38.385948 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:38.386550 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:38.386559 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:38.387032 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:38.387035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:38.387113 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:38.387666 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:38.387678 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:38.389813 (+ 2135us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.126:45513: Server oryx/127.16.233.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":94,"threads_started":1}
W20260810 22:34:38.390194 27567 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:34:38.384004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:58861 (local address 127.16.233.126:45513)
0810 22:34:38.384253 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:38.384257 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:38.384324 (+ 67us) server_negotiation.cc:407] Connection header received
0810 22:34:38.384461 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:38.384463 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:38.384508 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:38.384591 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:38.385186 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:38.385817 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:38.386674 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:38.386915 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:38.387000 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:38.390035 (+ 3035us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.65:58861: BlockingRecv error: recv got EOF from 127.16.233.65:58861 (error 108)
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":92,"threads_started":1}
Aug 10 22:34:38 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.66@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:34:38 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.66@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:34:38.582301 27568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:38.576404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:45513 (local address 127.16.233.66:49853)
0810 22:34:38.576928 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:38.576942 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:38.577085 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:38.577361 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:38.577366 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:38.577616 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:38.577725 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:38.577732 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:38.578594 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:38.578597 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:38.579165 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:38.579172 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:38.579663 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:38.579666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:38.579732 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:38.580293 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:38.580304 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:38.582169 (+ 1865us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.126:45513: Server oryx/127.16.233.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":94,"threads_started":1}
W20260810 22:34:38.582688 27567 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:34:38.576525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:49853 (local address 127.16.233.126:45513)
0810 22:34:38.576691 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:38.576695 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:38.577029 (+ 334us) server_negotiation.cc:407] Connection header received
0810 22:34:38.577204 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:38.577207 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:38.577253 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:38.577368 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:38.577904 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:38.578482 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:38.579290 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:38.579546 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:38.579641 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:38.582389 (+ 2748us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.66:49853: BlockingRecv error: recv got EOF from 127.16.233.66:49853 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 10 22:34:38 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.67@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:34:38 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.67@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:34:38.799207 27569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:38.792552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:45513 (local address 127.16.233.67:48375)
0810 22:34:38.792972 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:38.792986 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:38.793104 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:38.793390 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:38.793392 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:38.793590 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:38.793689 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:38.793695 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:38.794612 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:38.794616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:38.795315 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:38.795326 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:38.796017 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:38.796021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:38.796119 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:38.796733 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:38.796746 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:38.799029 (+ 2283us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.126:45513: Server oryx/127.16.233.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":140,"threads_started":1}
W20260810 22:34:38.799286 27567 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:34:38.792686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:48375 (local address 127.16.233.126:45513)
0810 22:34:38.792905 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:38.792909 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:38.793112 (+ 203us) server_negotiation.cc:407] Connection header received
0810 22:34:38.793178 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:38.793181 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:38.793245 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:38.793342 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:38.793854 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:38.794436 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:38.795468 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:38.795855 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:38.796000 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:38.799166 (+ 3166us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.67:48375: BlockingRecv error: recv got EOF from 127.16.233.67:48375 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 10 22:34:39 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.65@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:34:39 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.65@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:34:39.400873 27571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:39.390962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:45513 (local address 127.16.233.65:37667)
0810 22:34:39.391327 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:39.391342 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:39.391451 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:39.391843 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:39.391846 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:39.392052 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:39.392162 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:39.392168 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:39.393134 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:39.393138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:39.393713 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:39.393722 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:39.394259 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:39.394286 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:39.394356 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:39.398616 (+ 4260us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:39.398631 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:39.400649 (+ 2018us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.126:45513: Server oryx/127.16.233.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":112,"threads_started":1}
W20260810 22:34:39.401155 27572 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:34:39.391145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:37667 (local address 127.16.233.126:45513)
0810 22:34:39.391526 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:39.391537 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:39.391555 (+ 18us) server_negotiation.cc:407] Connection header received
0810 22:34:39.391603 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:39.391605 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:39.391652 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:39.391747 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:39.392318 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:39.392978 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:39.393850 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:39.394103 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:39.394224 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:39.400854 (+ 6630us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.65:37667: BlockingRecv error: recv got EOF from 127.16.233.65:37667 (error 108)
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":110,"threads_started":1}
Aug 10 22:34:39 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.66@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:34:39 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.66@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:34:39.589325 27573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:39.583210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:45513 (local address 127.16.233.66:44545)
0810 22:34:39.583560 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:39.583580 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:39.583721 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:39.584019 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:39.584022 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:39.584262 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:39.584358 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:39.584364 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:39.585204 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:39.585209 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:39.585808 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:39.585816 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:39.586321 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:39.586324 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:39.586414 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:39.586956 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:39.586968 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:39.589187 (+ 2219us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.126:45513: Server oryx/127.16.233.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":115,"threads_started":1}
W20260810 22:34:39.589537 27572 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:34:39.583283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:44545 (local address 127.16.233.126:45513)
0810 22:34:39.583511 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:39.583516 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:39.583661 (+ 145us) server_negotiation.cc:407] Connection header received
0810 22:34:39.583825 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:39.583829 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:39.583887 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:39.583994 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:39.584482 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:39.585093 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:39.585939 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:39.586199 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:39.586314 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:39.589369 (+ 3055us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.66:44545: BlockingRecv error: recv got EOF from 127.16.233.66:44545 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
Aug 10 22:34:39 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.67@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:34:39 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.67@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:34:39.805689 27574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:39.800090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:45513 (local address 127.16.233.67:43799)
0810 22:34:39.800407 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:39.800420 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:39.800518 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:39.800746 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:39.800748 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:39.801012 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:39.801112 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:39.801118 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:39.801905 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:39.801908 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:39.802490 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:39.802498 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:39.802960 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:39.802998 (+ 38us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:39.803102 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:39.803607 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:39.803619 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:39.805556 (+ 1937us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.126:45513: Server oryx/127.16.233.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":138,"threads_started":1}
W20260810 22:34:39.805811 27572 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:34:39.800151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:43799 (local address 127.16.233.126:45513)
0810 22:34:39.800330 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:39.800334 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:39.800485 (+ 151us) server_negotiation.cc:407] Connection header received
0810 22:34:39.800592 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:39.800594 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:39.800642 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:39.800713 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:39.801256 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:39.801802 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:39.802598 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:39.802831 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:39.802953 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:39.805693 (+ 2740us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.67:43799: BlockingRecv error: recv got EOF from 127.16.233.67:43799 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 10 22:34:40 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.65@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:34:40 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.65@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:34:40.408600 27575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:40.401966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:45513 (local address 127.16.233.65:60231)
0810 22:34:40.402292 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:40.402306 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:40.402426 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:40.402770 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:40.402773 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:40.402964 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:40.403067 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:40.403073 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:40.404068 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:40.404072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:40.404636 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:40.404644 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:40.405190 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:40.405194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:40.405285 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:40.406015 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:40.406029 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:40.408468 (+ 2439us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.126:45513: Server oryx/127.16.233.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":93,"threads_started":1}
W20260810 22:34:40.408926 27576 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:34:40.402064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:60231 (local address 127.16.233.126:45513)
0810 22:34:40.402312 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:40.402316 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:40.402393 (+ 77us) server_negotiation.cc:407] Connection header received
0810 22:34:40.402507 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:40.402509 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:40.402554 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:40.402637 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:40.403218 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:40.403929 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:40.404767 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:40.405039 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:40.405129 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:40.408716 (+ 3587us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.65:60231: BlockingRecv error: recv got EOF from 127.16.233.65:60231 (error 108)
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":63,"threads_started":1}
Aug 10 22:34:40 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.66@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:34:40 dist-test-slave-77v9 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.233.66@KRBTEST.COM for oryx/127.16.233.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:34:40.596839 27577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:40.590370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:45513 (local address 127.16.233.66:41187)
0810 22:34:40.590647 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:40.590661 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:40.590820 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:40.591142 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:40.591144 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:40.591352 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:40.591466 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:40.591472 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:40.592368 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:40.592371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:40.593079 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:40.593089 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:40.593609 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:40.593612 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:40.593677 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:40.594360 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:40.594373 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:40.596668 (+ 2295us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.233.126:45513: Server oryx/127.16.233.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":94,"threads_started":1}
W20260810 22:34:40.597132 27576 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:34:40.590463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:41187 (local address 127.16.233.126:45513)
0810 22:34:40.590652 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:40.590656 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:40.590745 (+ 89us) server_negotiation.cc:407] Connection header received
0810 22:34:40.590950 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:40.590952 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:40.591007 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:40.591085 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:40.591613 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:40.592213 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:40.593204 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:40.593475 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:40.593566 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:40.596999 (+ 3433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.233.66:41187: BlockingRecv error: recv got EOF from 127.16.233.66:41187 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260810 22:34:40.672919 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 27023
I20260810 22:34:40.683539 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 27197
I20260810 22:34:40.691540 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 27354
I20260810 22:34:40.699884 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 26943
2026-08-10T22:34:40Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18438 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-10T22:34:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:34:40Z Disabled control of system clock
I20260810 22:34:40.738185 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786401215402979-17317-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786401215402979-17317-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786401215402979-17317-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:39421
--webserver_interface=127.16.233.126
--webserver_port=0
--builtin_ntp_servers=127.16.233.84:33703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.126:39421
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260810 22:34:40.859622 27582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:34:40.859946 27582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:34:40.859982 27582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:34:40.864334 27582 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:34:40.864409 27582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:34:40.864424 27582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:34:40.864442 27582 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:34:40.864482 27582 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:34:40.869582 27582 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:33703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786401215402979-17317-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786401215402979-17317-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.126:39421
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:39421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786401215402979-17317-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:34:40.870831 27582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:34:40.872296 27582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 22:34:40.881706 27582 server_base.cc:1034] running on GCE node
W20260810 22:34:40.881819 27588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:40.881785 27589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:40.881707 27592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:40.882853 27582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:34:40.883980 27582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:34:40.885156 27582 hybrid_clock.cc:648] HybridClock initialized: now 1786401280885138 us; error 39 us; skew 500 ppm
I20260810 22:34:40.887346 27582 webserver.cc:460] Webserver started at http://127.16.233.126:44311/ using document root <none> and password file <none>
I20260810 22:34:40.887981 27582 fs_manager.cc:359] Metadata directory not provided
I20260810 22:34:40.888051 27582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:34:40.888278 27582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:34:40.890558 27582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786401215402979-17317-0/minicluster-data/master-0/data/instance:
uuid: "c4afd7f9b1e54b1a9dfe99a393b4a186"
format_stamp: "Formatted at 2026-08-10 22:34:40 on dist-test-slave-77v9"
server_key: "672f770fbe0eaf3eb488cf42dcbb6498"
server_key_iv: "78ca53f480bec3fef449d3c97ef714c2"
server_key_version: "encryptionkey@0"
I20260810 22:34:40.891188 27582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786401215402979-17317-0/minicluster-data/master-0/wal/instance:
uuid: "c4afd7f9b1e54b1a9dfe99a393b4a186"
format_stamp: "Formatted at 2026-08-10 22:34:40 on dist-test-slave-77v9"
server_key: "672f770fbe0eaf3eb488cf42dcbb6498"
server_key_iv: "78ca53f480bec3fef449d3c97ef714c2"
server_key_version: "encryptionkey@0"
I20260810 22:34:40.895457 27582 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 22:34:40.898243 27599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:40.899812 27582 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 22:34:40.900004 27582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786401215402979-17317-0/minicluster-data/master-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786401215402979-17317-0/minicluster-data/master-0/wal
uuid: "c4afd7f9b1e54b1a9dfe99a393b4a186"
format_stamp: "Formatted at 2026-08-10 22:34:40 on dist-test-slave-77v9"
server_key: "672f770fbe0eaf3eb488cf42dcbb6498"
server_key_iv: "78ca53f480bec3fef449d3c97ef714c2"
server_key_version: "encryptionkey@0"
I20260810 22:34:40.900163 27582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786401215402979-17317-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786401215402979-17317-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786401215402979-17317-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:34:40.913285 27582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:34:40.914165 27582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:34:40.914413 27582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:34:40.923017 27582 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.126:39421
I20260810 22:34:40.923089 27656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.126:39421 every 8 connection(s)
I20260810 22:34:40.924285 27582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
I20260810 22:34:40.925423 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 27582
I20260810 22:34:40.925583 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786401215402979-17317-0/minicluster-data/master-0/wal/instance
I20260810 22:34:40.926105 17317 external_mini_cluster.cc:1428] Setting key 4d055d25942485149ea2e568f6914eb2
I20260810 22:34:40.928929 27657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:34:40.935433 27657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4afd7f9b1e54b1a9dfe99a393b4a186: Bootstrap starting.
I20260810 22:34:40.936599 27661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:40.927871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43410 (local address 127.16.233.126:39421)
0810 22:34:40.928687 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:40.928712 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:40.928752 (+ 40us) server_negotiation.cc:407] Connection header received
0810 22:34:40.929591 (+ 839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:40.929609 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:40.930056 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:40.930445 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:40.930907 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:40.931920 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:40.932952 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:40.933461 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:40.933652 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:40.934654 (+ 1002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:40.934659 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:40.934673 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:34:40.935148 (+ 475us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:40.935212 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:34:40.935235 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:40.935357 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:40.935475 (+ 118us) server_negotiation.cc:299] Negotiation successful
0810 22:34:40.935789 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":170,"threads_started":1}
I20260810 22:34:40.938390 27657 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4afd7f9b1e54b1a9dfe99a393b4a186: Neither blocks nor log segments found. Creating new log.
I20260810 22:34:40.939376 27657 log.cc:824] T 00000000000000000000000000000000 P c4afd7f9b1e54b1a9dfe99a393b4a186: Log is configured to *not* fsync() on all Append() calls
I20260810 22:34:40.941941 27657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4afd7f9b1e54b1a9dfe99a393b4a186: No bootstrap required, opened a new log
I20260810 22:34:40.946157 27657 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4afd7f9b1e54b1a9dfe99a393b4a186 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4afd7f9b1e54b1a9dfe99a393b4a186" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39421 } }
I20260810 22:34:40.946473 27657 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4afd7f9b1e54b1a9dfe99a393b4a186 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:34:40.946557 27657 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4afd7f9b1e54b1a9dfe99a393b4a186 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4afd7f9b1e54b1a9dfe99a393b4a186, State: Initialized, Role: FOLLOWER
I20260810 22:34:40.947045 27657 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4afd7f9b1e54b1a9dfe99a393b4a186 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4afd7f9b1e54b1a9dfe99a393b4a186" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39421 } }
I20260810 22:34:40.947203 27657 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4afd7f9b1e54b1a9dfe99a393b4a186 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:34:40.947295 27657 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4afd7f9b1e54b1a9dfe99a393b4a186 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:34:40.947417 27657 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4afd7f9b1e54b1a9dfe99a393b4a186 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:34:40.948477 27657 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4afd7f9b1e54b1a9dfe99a393b4a186 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4afd7f9b1e54b1a9dfe99a393b4a186" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39421 } }
I20260810 22:34:40.948912 27657 leader_election.cc:302] T 00000000000000000000000000000000 P c4afd7f9b1e54b1a9dfe99a393b4a186 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4afd7f9b1e54b1a9dfe99a393b4a186; no voters:
I20260810 22:34:40.949301 27657 leader_election.cc:288] T 00000000000000000000000000000000 P c4afd7f9b1e54b1a9dfe99a393b4a186 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:34:40.949477 27663 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4afd7f9b1e54b1a9dfe99a393b4a186 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:34:40.949749 27663 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4afd7f9b1e54b1a9dfe99a393b4a186 [term 1 LEADER]: Becoming Leader. State: Replica: c4afd7f9b1e54b1a9dfe99a393b4a186, State: Running, Role: LEADER
I20260810 22:34:40.950147 27663 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4afd7f9b1e54b1a9dfe99a393b4a186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4afd7f9b1e54b1a9dfe99a393b4a186" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39421 } }
I20260810 22:34:40.950770 27657 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4afd7f9b1e54b1a9dfe99a393b4a186 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:34:40.951934 27664 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4afd7f9b1e54b1a9dfe99a393b4a186 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4afd7f9b1e54b1a9dfe99a393b4a186" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4afd7f9b1e54b1a9dfe99a393b4a186" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39421 } } }
I20260810 22:34:40.952092 27664 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4afd7f9b1e54b1a9dfe99a393b4a186 [sys.catalog]: This master's current role is: LEADER
I20260810 22:34:40.952428 27665 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4afd7f9b1e54b1a9dfe99a393b4a186 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4afd7f9b1e54b1a9dfe99a393b4a186. Latest consensus state: current_term: 1 leader_uuid: "c4afd7f9b1e54b1a9dfe99a393b4a186" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4afd7f9b1e54b1a9dfe99a393b4a186" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39421 } } }
I20260810 22:34:40.952525 27665 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4afd7f9b1e54b1a9dfe99a393b4a186 [sys.catalog]: This master's current role is: LEADER
I20260810 22:34:40.952517 27673 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:34:40.956552 27673 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:34:40.962754 27673 catalog_manager.cc:1347] Generated new cluster ID: e7e3ee6a0e64475482197220e6839209
I20260810 22:34:40.962864 27673 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:34:40.969753 27673 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:34:40.970933 27673 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:34:40.978868 27673 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4afd7f9b1e54b1a9dfe99a393b4a186: Generated new TSK 0
I20260810 22:34:40.979702 27673 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:34:40.984124 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.65:0
--local_ip_for_outbound_sockets=127.16.233.65
--webserver_interface=127.16.233.65
--webserver_port=0
--tserver_master_addrs=127.16.233.126:39421
--builtin_ntp_servers=127.16.233.84:33703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260810 22:34:41.110694 27687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:34:41.111027 27687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:34:41.111070 27687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:34:41.115876 27687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:34:41.115967 27687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:34:41.116024 27687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.65
I20260810 22:34:41.121187 27687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:33703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.233.65
--webserver_port=0
--tserver_master_addrs=127.16.233.126:39421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.65
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:34:41.122444 27687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:34:41.123951 27687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:34:41.134583 27697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:41.134604 27687 server_base.cc:1034] running on GCE node
W20260810 22:34:41.134579 27695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:41.134646 27694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:41.135456 27687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:34:41.136186 27687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:34:41.137378 27687 hybrid_clock.cc:648] HybridClock initialized: now 1786401281137361 us; error 33 us; skew 500 ppm
I20260810 22:34:41.139462 27687 webserver.cc:460] Webserver started at http://127.16.233.65:41083/ using document root <none> and password file <none>
I20260810 22:34:41.140074 27687 fs_manager.cc:359] Metadata directory not provided
I20260810 22:34:41.140173 27687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:34:41.140424 27687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:34:41.142706 27687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-0/data/instance:
uuid: "126068c245684b598e0e39835d825c50"
format_stamp: "Formatted at 2026-08-10 22:34:41 on dist-test-slave-77v9"
server_key: "fd95b6513932f43c02f1a0e97508058a"
server_key_iv: "a8c25f86d078743fbfb3994a28a12ad7"
server_key_version: "encryptionkey@0"
I20260810 22:34:41.143317 27687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-0/wal/instance:
uuid: "126068c245684b598e0e39835d825c50"
format_stamp: "Formatted at 2026-08-10 22:34:41 on dist-test-slave-77v9"
server_key: "fd95b6513932f43c02f1a0e97508058a"
server_key_iv: "a8c25f86d078743fbfb3994a28a12ad7"
server_key_version: "encryptionkey@0"
I20260810 22:34:41.147356 27687 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 22:34:41.150406 27704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:41.151961 27687 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 22:34:41.152088 27687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-0/wal
uuid: "126068c245684b598e0e39835d825c50"
format_stamp: "Formatted at 2026-08-10 22:34:41 on dist-test-slave-77v9"
server_key: "fd95b6513932f43c02f1a0e97508058a"
server_key_iv: "a8c25f86d078743fbfb3994a28a12ad7"
server_key_version: "encryptionkey@0"
I20260810 22:34:41.152264 27687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:34:41.162621 27687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:34:41.163316 27687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:34:41.163481 27687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:34:41.164047 27687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:34:41.165130 27687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:34:41.165186 27687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:41.165231 27687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:34:41.165251 27687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:41.174815 27687 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.65:44383
I20260810 22:34:41.174834 27835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.65:44383 every 8 connection(s)
I20260810 22:34:41.175822 27687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
I20260810 22:34:41.182660 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 27687
I20260810 22:34:41.182803 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-0/wal/instance
I20260810 22:34:41.183121 17317 external_mini_cluster.cc:1428] Setting key d7bf9c7b1318de1628db8ac35f222fa0
I20260810 22:34:41.185393 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.66:0
--local_ip_for_outbound_sockets=127.16.233.66
--webserver_interface=127.16.233.66
--webserver_port=0
--tserver_master_addrs=127.16.233.126:39421
--builtin_ntp_servers=127.16.233.84:33703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260810 22:34:41.186142 27661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:41.177876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:46009 (local address 127.16.233.126:39421)
0810 22:34:41.178026 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:41.178030 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:41.179173 (+ 1143us) server_negotiation.cc:407] Connection header received
0810 22:34:41.180740 (+ 1567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:41.180744 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:41.180866 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:41.180971 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:41.181912 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:41.182473 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:41.183868 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:41.184099 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:41.184215 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:41.185563 (+ 1348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:41.185571 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:41.185573 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:34:41.185586 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:41.185603 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:34:41.185611 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:41.185704 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:41.185902 (+ 198us) server_negotiation.cc:299] Negotiation successful
0810 22:34:41.185981 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 22:34:41.187060 27838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:41.178271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39421 (local address 127.16.233.65:46009)
0810 22:34:41.178983 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:41.179032 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:41.180499 (+ 1467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:41.181159 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:41.181166 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:41.181250 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:41.181761 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:41.181775 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:41.182622 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:41.182624 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:41.183748 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:41.183757 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:41.184262 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:41.184266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:41.184470 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:41.185263 (+ 793us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 22:34:41.185273 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:41.185331 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 22:34:41.185397 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:34:41.185710 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:34:41.185721 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:34:41.185727 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 22:34:41.185878 (+ 151us) client_negotiation.cc:240] Negotiation successful
0810 22:34:41.186176 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":119,"threads_started":1}
I20260810 22:34:41.188433 27836 heartbeater.cc:347] Connected to a master server at 127.16.233.126:39421
I20260810 22:34:41.188681 27836 heartbeater.cc:464] Registering TS with master...
I20260810 22:34:41.189314 27836 heartbeater.cc:511] Master 127.16.233.126:39421 requested a full tablet report, sending...
I20260810 22:34:41.190897 27619 ts_manager.cc:194] Registered new tserver with Master: 126068c245684b598e0e39835d825c50 (127.16.233.65:44383)
I20260810 22:34:41.192055 27619 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.233.65:46009
W20260810 22:34:41.302706 27839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:34:41.303025 27839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:34:41.303104 27839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:34:41.307368 27839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:34:41.307453 27839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:34:41.307545 27839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.66
I20260810 22:34:41.312327 27839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:33703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.233.66
--webserver_port=0
--tserver_master_addrs=127.16.233.126:39421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.66
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:34:41.313607 27839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:34:41.315044 27839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:34:41.324782 27846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:41.324872 27847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:41.324782 27849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:41.325165 27839 server_base.cc:1034] running on GCE node
I20260810 22:34:41.325556 27839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:34:41.326237 27839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:34:41.327430 27839 hybrid_clock.cc:648] HybridClock initialized: now 1786401281327396 us; error 50 us; skew 500 ppm
I20260810 22:34:41.329535 27839 webserver.cc:460] Webserver started at http://127.16.233.66:44929/ using document root <none> and password file <none>
I20260810 22:34:41.330161 27839 fs_manager.cc:359] Metadata directory not provided
I20260810 22:34:41.330263 27839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:34:41.330489 27839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:34:41.332629 27839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-1/data/instance:
uuid: "7030e623198947c3a4b7c4f7c7a87699"
format_stamp: "Formatted at 2026-08-10 22:34:41 on dist-test-slave-77v9"
server_key: "e5bb506bbcf2b51bc2fdcdc6432007c9"
server_key_iv: "2dfbae6253f02db6d79bf9b1abf061ec"
server_key_version: "encryptionkey@0"
I20260810 22:34:41.333276 27839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-1/wal/instance:
uuid: "7030e623198947c3a4b7c4f7c7a87699"
format_stamp: "Formatted at 2026-08-10 22:34:41 on dist-test-slave-77v9"
server_key: "e5bb506bbcf2b51bc2fdcdc6432007c9"
server_key_iv: "2dfbae6253f02db6d79bf9b1abf061ec"
server_key_version: "encryptionkey@0"
I20260810 22:34:41.337175 27839 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 22:34:41.340004 27857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:41.341403 27839 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:34:41.341545 27839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-1/wal
uuid: "7030e623198947c3a4b7c4f7c7a87699"
format_stamp: "Formatted at 2026-08-10 22:34:41 on dist-test-slave-77v9"
server_key: "e5bb506bbcf2b51bc2fdcdc6432007c9"
server_key_iv: "2dfbae6253f02db6d79bf9b1abf061ec"
server_key_version: "encryptionkey@0"
I20260810 22:34:41.341670 27839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:34:41.360541 27839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:34:41.361397 27839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:34:41.361617 27839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:34:41.362287 27839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:34:41.363278 27839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:34:41.363353 27839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:41.363421 27839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:34:41.363466 27839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:41.373034 27839 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.66:38621
I20260810 22:34:41.373063 27987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.66:38621 every 8 connection(s)
I20260810 22:34:41.373958 27839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
I20260810 22:34:41.383287 27661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:41.375903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:55301 (local address 127.16.233.126:39421)
0810 22:34:41.376052 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:41.376056 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:41.377141 (+ 1085us) server_negotiation.cc:407] Connection header received
0810 22:34:41.378562 (+ 1421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:41.378565 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:41.378626 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:41.378738 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:41.379601 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:41.380136 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:41.381342 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:41.381571 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:41.381685 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:41.382787 (+ 1102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:41.382790 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:41.382792 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:34:41.382805 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:41.382824 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:34:41.382831 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:41.382916 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:41.383065 (+ 149us) server_negotiation.cc:299] Negotiation successful
0810 22:34:41.383135 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 22:34:41.383817 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 27839
I20260810 22:34:41.383949 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-1/wal/instance
I20260810 22:34:41.384255 17317 external_mini_cluster.cc:1428] Setting key cf917a4196d89f31e8d7e7ec690a2de3
I20260810 22:34:41.384377 27990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:41.376296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39421 (local address 127.16.233.66:55301)
0810 22:34:41.377001 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:41.377043 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:41.378358 (+ 1315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:41.378905 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:41.378911 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:41.378990 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:41.379460 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:41.379471 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:41.380275 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:41.380278 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:41.381219 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:41.381229 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:41.381728 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:41.381732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:41.381890 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:41.382525 (+ 635us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 22:34:41.382526 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:41.382565 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 22:34:41.382625 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:34:41.382918 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:34:41.382929 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:34:41.382937 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 22:34:41.383062 (+ 125us) client_negotiation.cc:240] Negotiation successful
0810 22:34:41.383334 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":123,"threads_started":1}
I20260810 22:34:41.386297 27988 heartbeater.cc:347] Connected to a master server at 127.16.233.126:39421
I20260810 22:34:41.386566 27988 heartbeater.cc:464] Registering TS with master...
I20260810 22:34:41.387001 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.67:0
--local_ip_for_outbound_sockets=127.16.233.67
--webserver_interface=127.16.233.67
--webserver_port=0
--tserver_master_addrs=127.16.233.126:39421
--builtin_ntp_servers=127.16.233.84:33703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260810 22:34:41.387295 27988 heartbeater.cc:511] Master 127.16.233.126:39421 requested a full tablet report, sending...
I20260810 22:34:41.388401 27619 ts_manager.cc:194] Registered new tserver with Master: 7030e623198947c3a4b7c4f7c7a87699 (127.16.233.66:38621)
I20260810 22:34:41.389219 27619 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.233.66:55301
W20260810 22:34:41.507645 27993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:34:41.507942 27993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:34:41.507985 27993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:34:41.512137 27993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:34:41.512218 27993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:34:41.512276 27993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.67
I20260810 22:34:41.517227 27993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:33703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.233.67
--webserver_port=0
--tserver_master_addrs=127.16.233.126:39421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.67
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:34:41.518437 27993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:34:41.519951 27993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:34:41.530432 28000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:41.530514 27993 server_base.cc:1034] running on GCE node
W20260810 22:34:41.530444 28002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:41.530465 27998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:41.531129 27993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:34:41.531813 27993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:34:41.532994 27993 hybrid_clock.cc:648] HybridClock initialized: now 1786401281532972 us; error 32 us; skew 500 ppm
I20260810 22:34:41.535162 27993 webserver.cc:460] Webserver started at http://127.16.233.67:37945/ using document root <none> and password file <none>
I20260810 22:34:41.535847 27993 fs_manager.cc:359] Metadata directory not provided
I20260810 22:34:41.535925 27993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:34:41.536185 27993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:34:41.538744 27993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-2/data/instance:
uuid: "e8f9d6bf1f14425a832ab0395f5472d3"
format_stamp: "Formatted at 2026-08-10 22:34:41 on dist-test-slave-77v9"
server_key: "df3fbebc21305b6dfb80165a328d4d41"
server_key_iv: "f7b077d19df519231c8f8695c7c4b0b2"
server_key_version: "encryptionkey@0"
I20260810 22:34:41.539448 27993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-2/wal/instance:
uuid: "e8f9d6bf1f14425a832ab0395f5472d3"
format_stamp: "Formatted at 2026-08-10 22:34:41 on dist-test-slave-77v9"
server_key: "df3fbebc21305b6dfb80165a328d4d41"
server_key_iv: "f7b077d19df519231c8f8695c7c4b0b2"
server_key_version: "encryptionkey@0"
I20260810 22:34:41.543915 27993 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260810 22:34:41.546988 28010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:41.548377 27993 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 22:34:41.548520 27993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-2/wal
uuid: "e8f9d6bf1f14425a832ab0395f5472d3"
format_stamp: "Formatted at 2026-08-10 22:34:41 on dist-test-slave-77v9"
server_key: "df3fbebc21305b6dfb80165a328d4d41"
server_key_iv: "f7b077d19df519231c8f8695c7c4b0b2"
server_key_version: "encryptionkey@0"
I20260810 22:34:41.548642 27993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:34:41.568516 27993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:34:41.569350 27993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:34:41.569571 27993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:34:41.570258 27993 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:34:41.571329 27993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:34:41.571378 27993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:41.571420 27993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:34:41.571473 27993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:41.581161 27993 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.67:40805
I20260810 22:34:41.581182 28135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.67:40805 every 8 connection(s)
I20260810 22:34:41.582149 27993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
I20260810 22:34:41.586241 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 27993
I20260810 22:34:41.586370 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-2/wal/instance
I20260810 22:34:41.586630 17317 external_mini_cluster.cc:1428] Setting key f51594960b1a7147d1aa3c7018a7676b
I20260810 22:34:41.591641 27661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:41.584059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:47377 (local address 127.16.233.126:39421)
0810 22:34:41.584217 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:41.584221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:41.585369 (+ 1148us) server_negotiation.cc:407] Connection header received
0810 22:34:41.586892 (+ 1523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:41.586895 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:41.586957 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:41.587071 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:41.587976 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:41.588512 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:41.589721 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:41.589950 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:41.590064 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:41.591119 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:41.591123 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:41.591126 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:34:41.591147 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:41.591166 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:34:41.591174 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:41.591250 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:41.591435 (+ 185us) server_negotiation.cc:299] Negotiation successful
0810 22:34:41.591499 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 22:34:41.592489 28140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:41.584495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39421 (local address 127.16.233.67:47377)
0810 22:34:41.585214 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:41.585257 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:41.586651 (+ 1394us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:41.587252 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:41.587259 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:41.587349 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:41.587834 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:41.587847 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:41.588654 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:41.588657 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:41.589591 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:41.589600 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:41.590104 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:41.590108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:41.590270 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:41.590843 (+ 573us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 22:34:41.590844 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:41.590884 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 22:34:41.590947 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:34:41.591264 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:34:41.591276 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:34:41.591286 (+ 10us) client_negotiation.cc:769] Sending connection context
0810 22:34:41.591435 (+ 149us) client_negotiation.cc:240] Negotiation successful
0810 22:34:41.591712 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":112,"threads_started":1}
I20260810 22:34:41.593676 28136 heartbeater.cc:347] Connected to a master server at 127.16.233.126:39421
I20260810 22:34:41.593971 28136 heartbeater.cc:464] Registering TS with master...
I20260810 22:34:41.594549 28136 heartbeater.cc:511] Master 127.16.233.126:39421 requested a full tablet report, sending...
I20260810 22:34:41.595611 27619 ts_manager.cc:194] Registered new tserver with Master: e8f9d6bf1f14425a832ab0395f5472d3 (127.16.233.67:40805)
I20260810 22:34:41.596259 27619 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.233.67:47377
I20260810 22:34:41.600963 17317 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 22:34:41.604425 27661 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:34:41.603059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43416 (local address 127.16.233.126:39421)
0810 22:34:41.603205 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:41.603208 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:41.603464 (+ 256us) server_negotiation.cc:407] Connection header received
0810 22:34:41.603561 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:41.603564 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:41.603606 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:41.603684 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:41.604222 (+ 538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43416: BlockingRecv error: recv got EOF from 127.0.0.1:43416 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 22:34:41.604540 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 27687
I20260810 22:34:41.616971 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 27839
I20260810 22:34:41.628016 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 27993
I20260810 22:34:41.637466 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 27582
2026-08-10T22:34:41Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (934 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-10T22:34:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:34:41Z Disabled control of system clock
I20260810 22:34:41.673869 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786401215402979-17317-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786401215402979-17317-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786401215402979-17317-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:41859
--webserver_interface=127.16.233.126
--webserver_port=0
--builtin_ntp_servers=127.16.233.84:35951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.126:41859
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 22:34:41.791025 28150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:34:41.791316 28150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:34:41.791357 28150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:34:41.795452 28150 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:34:41.795518 28150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:34:41.795535 28150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:34:41.795554 28150 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:34:41.795569 28150 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:34:41.800506 28150 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:35951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786401215402979-17317-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786401215402979-17317-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.126:41859
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:41859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786401215402979-17317-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:34:41.801704 28150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:34:41.803114 28150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:34:41.812103 28160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:41.812104 28157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:41.812072 28158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:41.812383 28150 server_base.cc:1034] running on GCE node
I20260810 22:34:41.813139 28150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:34:41.814283 28150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:34:41.815452 28150 hybrid_clock.cc:648] HybridClock initialized: now 1786401281815436 us; error 31 us; skew 500 ppm
I20260810 22:34:41.817499 28150 webserver.cc:460] Webserver started at http://127.16.233.126:33593/ using document root <none> and password file <none>
I20260810 22:34:41.818128 28150 fs_manager.cc:359] Metadata directory not provided
I20260810 22:34:41.818195 28150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:34:41.818483 28150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:34:41.820907 28150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786401215402979-17317-0/minicluster-data/master-0/data/instance:
uuid: "ef15e6ddf67440c5a7890c3b5ea8e945"
format_stamp: "Formatted at 2026-08-10 22:34:41 on dist-test-slave-77v9"
server_key: "ab6335444edc6f071522dbaa5f447656"
server_key_iv: "c0a5b80a2d0187dc110b93ce3d32df74"
server_key_version: "encryptionkey@0"
I20260810 22:34:41.821553 28150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786401215402979-17317-0/minicluster-data/master-0/wal/instance:
uuid: "ef15e6ddf67440c5a7890c3b5ea8e945"
format_stamp: "Formatted at 2026-08-10 22:34:41 on dist-test-slave-77v9"
server_key: "ab6335444edc6f071522dbaa5f447656"
server_key_iv: "c0a5b80a2d0187dc110b93ce3d32df74"
server_key_version: "encryptionkey@0"
I20260810 22:34:41.825488 28150 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 22:34:41.828284 28166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:41.829954 28150 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 22:34:41.830147 28150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786401215402979-17317-0/minicluster-data/master-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786401215402979-17317-0/minicluster-data/master-0/wal
uuid: "ef15e6ddf67440c5a7890c3b5ea8e945"
format_stamp: "Formatted at 2026-08-10 22:34:41 on dist-test-slave-77v9"
server_key: "ab6335444edc6f071522dbaa5f447656"
server_key_iv: "c0a5b80a2d0187dc110b93ce3d32df74"
server_key_version: "encryptionkey@0"
I20260810 22:34:41.830367 28150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786401215402979-17317-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786401215402979-17317-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786401215402979-17317-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:34:41.835994 28150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:34:41.836728 28150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:34:41.836990 28150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:34:41.845575 28150 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.126:41859
I20260810 22:34:41.845597 28223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.126:41859 every 8 connection(s)
I20260810 22:34:41.846743 28150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
I20260810 22:34:41.850437 28224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:34:41.850893 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 28150
I20260810 22:34:41.851014 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786401215402979-17317-0/minicluster-data/master-0/wal/instance
I20260810 22:34:41.851285 17317 external_mini_cluster.cc:1428] Setting key 81491f6e64f6452d3f08f180756e5c7c
I20260810 22:34:41.857139 28224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef15e6ddf67440c5a7890c3b5ea8e945: Bootstrap starting.
I20260810 22:34:41.857412 28227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:41.852869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53892 (local address 127.16.233.126:41859)
0810 22:34:41.853528 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:41.853545 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:41.853581 (+ 36us) server_negotiation.cc:407] Connection header received
0810 22:34:41.854297 (+ 716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:41.854314 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:41.854676 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:41.854968 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:41.855895 (+ 927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:41.855900 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:41.855911 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:34:41.855921 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:41.855975 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:34:41.855991 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:41.856104 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:41.856250 (+ 146us) server_negotiation.cc:299] Negotiation successful
0810 22:34:41.856527 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":139,"threads_started":1}
I20260810 22:34:41.859825 28224 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef15e6ddf67440c5a7890c3b5ea8e945: Neither blocks nor log segments found. Creating new log.
I20260810 22:34:41.860601 28224 log.cc:824] T 00000000000000000000000000000000 P ef15e6ddf67440c5a7890c3b5ea8e945: Log is configured to *not* fsync() on all Append() calls
I20260810 22:34:41.862740 28224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef15e6ddf67440c5a7890c3b5ea8e945: No bootstrap required, opened a new log
I20260810 22:34:41.866921 28224 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef15e6ddf67440c5a7890c3b5ea8e945 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef15e6ddf67440c5a7890c3b5ea8e945" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 41859 } }
I20260810 22:34:41.867195 28224 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef15e6ddf67440c5a7890c3b5ea8e945 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:34:41.867276 28224 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef15e6ddf67440c5a7890c3b5ea8e945 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef15e6ddf67440c5a7890c3b5ea8e945, State: Initialized, Role: FOLLOWER
I20260810 22:34:41.867712 28224 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef15e6ddf67440c5a7890c3b5ea8e945 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef15e6ddf67440c5a7890c3b5ea8e945" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 41859 } }
I20260810 22:34:41.867874 28224 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef15e6ddf67440c5a7890c3b5ea8e945 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:34:41.867964 28224 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef15e6ddf67440c5a7890c3b5ea8e945 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:34:41.868081 28224 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef15e6ddf67440c5a7890c3b5ea8e945 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:34:41.869179 28224 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef15e6ddf67440c5a7890c3b5ea8e945 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef15e6ddf67440c5a7890c3b5ea8e945" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 41859 } }
I20260810 22:34:41.869524 28224 leader_election.cc:302] T 00000000000000000000000000000000 P ef15e6ddf67440c5a7890c3b5ea8e945 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef15e6ddf67440c5a7890c3b5ea8e945; no voters:
I20260810 22:34:41.869864 28224 leader_election.cc:288] T 00000000000000000000000000000000 P ef15e6ddf67440c5a7890c3b5ea8e945 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:34:41.870023 28229 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef15e6ddf67440c5a7890c3b5ea8e945 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:34:41.870303 28229 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef15e6ddf67440c5a7890c3b5ea8e945 [term 1 LEADER]: Becoming Leader. State: Replica: ef15e6ddf67440c5a7890c3b5ea8e945, State: Running, Role: LEADER
I20260810 22:34:41.870931 28229 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef15e6ddf67440c5a7890c3b5ea8e945 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef15e6ddf67440c5a7890c3b5ea8e945" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 41859 } }
I20260810 22:34:41.871268 28224 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef15e6ddf67440c5a7890c3b5ea8e945 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:34:41.872629 28231 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef15e6ddf67440c5a7890c3b5ea8e945 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef15e6ddf67440c5a7890c3b5ea8e945. Latest consensus state: current_term: 1 leader_uuid: "ef15e6ddf67440c5a7890c3b5ea8e945" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef15e6ddf67440c5a7890c3b5ea8e945" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 41859 } } }
I20260810 22:34:41.872776 28231 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef15e6ddf67440c5a7890c3b5ea8e945 [sys.catalog]: This master's current role is: LEADER
I20260810 22:34:41.873104 28230 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef15e6ddf67440c5a7890c3b5ea8e945 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef15e6ddf67440c5a7890c3b5ea8e945" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef15e6ddf67440c5a7890c3b5ea8e945" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 41859 } } }
I20260810 22:34:41.873200 28230 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef15e6ddf67440c5a7890c3b5ea8e945 [sys.catalog]: This master's current role is: LEADER
W20260810 22:34:41.875736 28250 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ef15e6ddf67440c5a7890c3b5ea8e945: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 22:34:41.875804 28250 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 22:34:41.875898 28244 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:34:41.876533 28244 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:34:41.881549 28244 catalog_manager.cc:1347] Generated new cluster ID: 21f2781f0ef9491d9e6cf47ef6223171
I20260810 22:34:41.881616 28244 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:34:41.907109 28244 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:34:41.907470 28244 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:34:41.915108 28244 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef15e6ddf67440c5a7890c3b5ea8e945: Generated new TSK 0
I20260810 22:34:41.915789 28244 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:34:41.929306 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.65:0
--local_ip_for_outbound_sockets=127.16.233.65
--webserver_interface=127.16.233.65
--webserver_port=0
--tserver_master_addrs=127.16.233.126:41859
--builtin_ntp_servers=127.16.233.84:35951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 22:34:42.045637 28255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:34:42.045992 28255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:34:42.046067 28255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:34:42.050197 28255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:34:42.050252 28255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:34:42.050344 28255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.65
I20260810 22:34:42.055217 28255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:35951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.233.65
--webserver_port=0
--tserver_master_addrs=127.16.233.126:41859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.65
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:34:42.056406 28255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:34:42.057859 28255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:34:42.067869 28264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:42.067874 28262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:42.067970 28255 server_base.cc:1034] running on GCE node
W20260810 22:34:42.067982 28260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:42.068662 28255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:34:42.069375 28255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:34:42.070566 28255 hybrid_clock.cc:648] HybridClock initialized: now 1786401282070546 us; error 34 us; skew 500 ppm
I20260810 22:34:42.072773 28255 webserver.cc:460] Webserver started at http://127.16.233.65:33021/ using document root <none> and password file <none>
I20260810 22:34:42.073513 28255 fs_manager.cc:359] Metadata directory not provided
I20260810 22:34:42.073613 28255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:34:42.073839 28255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:34:42.075975 28255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-0/data/instance:
uuid: "6f6c1ec6555f4cd39543c6ca8fe90c78"
format_stamp: "Formatted at 2026-08-10 22:34:42 on dist-test-slave-77v9"
server_key: "f1e5f93caceb6f8eaf0d5019bbda60e8"
server_key_iv: "a26dc4f97a751f3bd82fe0a69bec2f21"
server_key_version: "encryptionkey@0"
I20260810 22:34:42.076634 28255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-0/wal/instance:
uuid: "6f6c1ec6555f4cd39543c6ca8fe90c78"
format_stamp: "Formatted at 2026-08-10 22:34:42 on dist-test-slave-77v9"
server_key: "f1e5f93caceb6f8eaf0d5019bbda60e8"
server_key_iv: "a26dc4f97a751f3bd82fe0a69bec2f21"
server_key_version: "encryptionkey@0"
I20260810 22:34:42.080489 28255 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:34:42.083261 28270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:42.084594 28255 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 22:34:42.084758 28255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-0/wal
uuid: "6f6c1ec6555f4cd39543c6ca8fe90c78"
format_stamp: "Formatted at 2026-08-10 22:34:42 on dist-test-slave-77v9"
server_key: "f1e5f93caceb6f8eaf0d5019bbda60e8"
server_key_iv: "a26dc4f97a751f3bd82fe0a69bec2f21"
server_key_version: "encryptionkey@0"
I20260810 22:34:42.084980 28255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:34:42.089807 28255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:34:42.090442 28255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:34:42.090638 28255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:34:42.091198 28255 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:34:42.092120 28255 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:34:42.092192 28255 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:42.092274 28255 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:34:42.092319 28255 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:42.101186 28255 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.65:45301
I20260810 22:34:42.101208 28399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.65:45301 every 8 connection(s)
I20260810 22:34:42.102121 28255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
I20260810 22:34:42.106925 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 28255
I20260810 22:34:42.107060 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-0/wal/instance
I20260810 22:34:42.107360 17317 external_mini_cluster.cc:1428] Setting key dbcfd31686c145a485277a3391f04ac2
I20260810 22:34:42.108600 28227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:42.103990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:36871 (local address 127.16.233.126:41859)
0810 22:34:42.104139 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:42.104143 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:42.105331 (+ 1188us) server_negotiation.cc:407] Connection header received
0810 22:34:42.106700 (+ 1369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:42.106704 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:42.106752 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:42.106856 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:42.108129 (+ 1273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:42.108133 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:42.108135 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:34:42.108147 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:42.108167 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:34:42.108175 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:42.108275 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:42.108420 (+ 145us) server_negotiation.cc:299] Negotiation successful
0810 22:34:42.108469 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 22:34:42.109555 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.66:0
--local_ip_for_outbound_sockets=127.16.233.66
--webserver_interface=127.16.233.66
--webserver_port=0
--tserver_master_addrs=127.16.233.126:41859
--builtin_ntp_servers=127.16.233.84:35951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260810 22:34:42.109817 28404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:42.104379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:41859 (local address 127.16.233.65:36871)
0810 22:34:42.105172 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:42.105214 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:42.106473 (+ 1259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:42.107032 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:42.107062 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:42.107179 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:42.107847 (+ 668us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 22:34:42.107849 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:42.107898 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 22:34:42.107964 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:34:42.108261 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:34:42.108270 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:34:42.108275 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 22:34:42.108468 (+ 193us) client_negotiation.cc:240] Negotiation successful
0810 22:34:42.108802 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":116,"threads_started":1}
I20260810 22:34:42.111095 28401 heartbeater.cc:347] Connected to a master server at 127.16.233.126:41859
I20260810 22:34:42.111397 28401 heartbeater.cc:464] Registering TS with master...
I20260810 22:34:42.111894 28401 heartbeater.cc:511] Master 127.16.233.126:41859 requested a full tablet report, sending...
I20260810 22:34:42.113824 28184 ts_manager.cc:194] Registered new tserver with Master: 6f6c1ec6555f4cd39543c6ca8fe90c78 (127.16.233.65:45301)
W20260810 22:34:42.227896 28406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:34:42.228158 28406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:34:42.228199 28406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:34:42.232156 28406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:34:42.232216 28406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:34:42.232275 28406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.66
I20260810 22:34:42.237144 28406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:35951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.233.66
--webserver_port=0
--tserver_master_addrs=127.16.233.126:41859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.66
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:34:42.238189 28406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:34:42.239558 28406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:34:42.249383 28412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:42.249397 28413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:42.249493 28415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:42.249442 28406 server_base.cc:1034] running on GCE node
I20260810 22:34:42.250023 28406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:34:42.250666 28406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:34:42.251827 28406 hybrid_clock.cc:648] HybridClock initialized: now 1786401282251795 us; error 46 us; skew 500 ppm
I20260810 22:34:42.253830 28406 webserver.cc:460] Webserver started at http://127.16.233.66:43551/ using document root <none> and password file <none>
I20260810 22:34:42.254410 28406 fs_manager.cc:359] Metadata directory not provided
I20260810 22:34:42.254472 28406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:34:42.254637 28406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:34:42.256672 28406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-1/data/instance:
uuid: "c44902c6ed25483b817cb0eeaac99677"
format_stamp: "Formatted at 2026-08-10 22:34:42 on dist-test-slave-77v9"
server_key: "e20e908c5ab99bcfcb1f558543e78979"
server_key_iv: "eea576e25acbc09726681fb9bdd15344"
server_key_version: "encryptionkey@0"
I20260810 22:34:42.257321 28406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-1/wal/instance:
uuid: "c44902c6ed25483b817cb0eeaac99677"
format_stamp: "Formatted at 2026-08-10 22:34:42 on dist-test-slave-77v9"
server_key: "e20e908c5ab99bcfcb1f558543e78979"
server_key_iv: "eea576e25acbc09726681fb9bdd15344"
server_key_version: "encryptionkey@0"
I20260810 22:34:42.261020 28406 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:34:42.263660 28422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:42.265043 28406 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:34:42.265142 28406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-1/wal
uuid: "c44902c6ed25483b817cb0eeaac99677"
format_stamp: "Formatted at 2026-08-10 22:34:42 on dist-test-slave-77v9"
server_key: "e20e908c5ab99bcfcb1f558543e78979"
server_key_iv: "eea576e25acbc09726681fb9bdd15344"
server_key_version: "encryptionkey@0"
I20260810 22:34:42.265280 28406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:34:42.270005 28406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:34:42.270926 28406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:34:42.271121 28406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:34:42.271673 28406 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:34:42.272914 28406 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:34:42.272979 28406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:42.273029 28406 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:34:42.273061 28406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:42.282076 28406 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.66:41403
I20260810 22:34:42.282158 28550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.66:41403 every 8 connection(s)
I20260810 22:34:42.283051 28406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
I20260810 22:34:42.287604 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 28406
I20260810 22:34:42.287767 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-1/wal/instance
I20260810 22:34:42.288084 17317 external_mini_cluster.cc:1428] Setting key c824baa67093b1e5e1357faf69cda353
I20260810 22:34:42.289482 28227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:42.284877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:38679 (local address 127.16.233.126:41859)
0810 22:34:42.285027 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:42.285031 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:42.286085 (+ 1054us) server_negotiation.cc:407] Connection header received
0810 22:34:42.287468 (+ 1383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:42.287471 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:42.287519 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:42.287627 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:42.288954 (+ 1327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:42.288957 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:42.288960 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:34:42.288973 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:42.288998 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:34:42.289007 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:42.289095 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:42.289300 (+ 205us) server_negotiation.cc:299] Negotiation successful
0810 22:34:42.289347 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 22:34:42.290501 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.67:0
--local_ip_for_outbound_sockets=127.16.233.67
--webserver_interface=127.16.233.67
--webserver_port=0
--tserver_master_addrs=127.16.233.126:41859
--builtin_ntp_servers=127.16.233.84:35951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260810 22:34:42.290614 28553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:42.285307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:41859 (local address 127.16.233.66:38679)
0810 22:34:42.285939 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:42.285979 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:42.287232 (+ 1253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:42.287809 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:42.287816 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:42.287942 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:42.288615 (+ 673us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 22:34:42.288617 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:42.288670 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 22:34:42.288760 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:34:42.289086 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:34:42.289094 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:34:42.289099 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 22:34:42.289303 (+ 204us) client_negotiation.cc:240] Negotiation successful
0810 22:34:42.289518 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":110,"threads_started":1}
I20260810 22:34:42.291769 28551 heartbeater.cc:347] Connected to a master server at 127.16.233.126:41859
I20260810 22:34:42.292033 28551 heartbeater.cc:464] Registering TS with master...
I20260810 22:34:42.292511 28551 heartbeater.cc:511] Master 127.16.233.126:41859 requested a full tablet report, sending...
I20260810 22:34:42.293701 28184 ts_manager.cc:194] Registered new tserver with Master: c44902c6ed25483b817cb0eeaac99677 (127.16.233.66:41403)
W20260810 22:34:42.406405 28554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:34:42.406702 28554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:34:42.406746 28554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:34:42.411002 28554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:34:42.411074 28554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:34:42.411134 28554 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.67
I20260810 22:34:42.415964 28554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:35951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.233.67
--webserver_port=0
--tserver_master_addrs=127.16.233.126:41859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.67
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:34:42.417078 28554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:34:42.418601 28554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:34:42.428323 28559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:42.428329 28562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:42.428339 28564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:42.428490 28554 server_base.cc:1034] running on GCE node
I20260810 22:34:42.429077 28554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:34:42.429785 28554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:34:42.430994 28554 hybrid_clock.cc:648] HybridClock initialized: now 1786401282430963 us; error 36 us; skew 500 ppm
I20260810 22:34:42.432889 28554 webserver.cc:460] Webserver started at http://127.16.233.67:44061/ using document root <none> and password file <none>
I20260810 22:34:42.433486 28554 fs_manager.cc:359] Metadata directory not provided
I20260810 22:34:42.433581 28554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:34:42.433830 28554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:34:42.436031 28554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-2/data/instance:
uuid: "27d0151e8868455dbbc123db970c29d4"
format_stamp: "Formatted at 2026-08-10 22:34:42 on dist-test-slave-77v9"
server_key: "bc0fcc1b8401fbf13598ffb1395dd4c9"
server_key_iv: "4115fc36a9e3cad21571f3828309706e"
server_key_version: "encryptionkey@0"
I20260810 22:34:42.436602 28554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-2/wal/instance:
uuid: "27d0151e8868455dbbc123db970c29d4"
format_stamp: "Formatted at 2026-08-10 22:34:42 on dist-test-slave-77v9"
server_key: "bc0fcc1b8401fbf13598ffb1395dd4c9"
server_key_iv: "4115fc36a9e3cad21571f3828309706e"
server_key_version: "encryptionkey@0"
I20260810 22:34:42.440351 28554 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 22:34:42.442903 28571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:42.444113 28554 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:34:42.444258 28554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-2/wal
uuid: "27d0151e8868455dbbc123db970c29d4"
format_stamp: "Formatted at 2026-08-10 22:34:42 on dist-test-slave-77v9"
server_key: "bc0fcc1b8401fbf13598ffb1395dd4c9"
server_key_iv: "4115fc36a9e3cad21571f3828309706e"
server_key_version: "encryptionkey@0"
I20260810 22:34:42.444381 28554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:34:42.449093 28554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:34:42.449712 28554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:34:42.449905 28554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:34:42.450454 28554 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:34:42.451370 28554 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:34:42.451440 28554 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:42.451510 28554 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:34:42.451551 28554 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:42.460788 28554 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.67:44591
I20260810 22:34:42.460871 28708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.67:44591 every 8 connection(s)
I20260810 22:34:42.461854 28554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
I20260810 22:34:42.468638 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 28554
I20260810 22:34:42.468792 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786401215402979-17317-0/minicluster-data/ts-2/wal/instance
I20260810 22:34:42.468786 28227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:42.463741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:49315 (local address 127.16.233.126:41859)
0810 22:34:42.463909 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:42.463914 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:42.465306 (+ 1392us) server_negotiation.cc:407] Connection header received
0810 22:34:42.466687 (+ 1381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:42.466690 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:42.466728 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:42.466810 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:42.468117 (+ 1307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:42.468121 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:42.468123 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:34:42.468137 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:42.468159 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:34:42.468168 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:42.468387 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:42.468574 (+ 187us) server_negotiation.cc:299] Negotiation successful
0810 22:34:42.468629 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 22:34:42.469165 17317 external_mini_cluster.cc:1428] Setting key 9625e631ae2bd1db1fb2d59b1377fee3
I20260810 22:34:42.469973 28711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:42.464288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:41859 (local address 127.16.233.67:49315)
0810 22:34:42.465142 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:42.465187 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:42.466491 (+ 1304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:42.466978 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:42.466985 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:42.467077 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:42.467811 (+ 734us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 22:34:42.467813 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:42.467867 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 22:34:42.467940 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:34:42.468380 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:34:42.468390 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:34:42.468398 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 22:34:42.468605 (+ 207us) client_negotiation.cc:240] Negotiation successful
0810 22:34:42.468977 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":141,"threads_started":1}
I20260810 22:34:42.471190 28709 heartbeater.cc:347] Connected to a master server at 127.16.233.126:41859
I20260810 22:34:42.471499 28709 heartbeater.cc:464] Registering TS with master...
I20260810 22:34:42.471977 28709 heartbeater.cc:511] Master 127.16.233.126:41859 requested a full tablet report, sending...
I20260810 22:34:42.473165 28184 ts_manager.cc:194] Registered new tserver with Master: 27d0151e8868455dbbc123db970c29d4 (127.16.233.67:44591)
I20260810 22:34:42.483985 17317 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 22:34:42.487172 28227 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:34:42.486152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53902 (local address 127.16.233.126:41859)
0810 22:34:42.486332 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:42.486338 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:42.486501 (+ 163us) server_negotiation.cc:407] Connection header received
0810 22:34:42.486674 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:42.486681 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:42.486718 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:42.486801 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:42.487057 (+ 256us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53902: BlockingRecv error: recv got EOF from 127.0.0.1:53902 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 22:34:42.487705 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 28255
I20260810 22:34:42.495384 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 28406
I20260810 22:34:42.503108 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 28554
I20260810 22:34:42.511876 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 28150
2026-08-10T22:34:42Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (873 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:34:42 dist-test-slave-77v9 krb5kdc[28723](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:34:42 dist-test-slave-77v9 krb5kdc[28723](info): set up 2 sockets
Aug 10 22:34:42 dist-test-slave-77v9 krb5kdc[28723](info): commencing operation
krb5kdc: starting...
W20260810 22:34:43.673509 17317 mini_kdc.cc:121] Time spent starting KDC: real 1.138s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Aug 10 22:34:43 dist-test-slave-77v9 krb5kdc[28723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:34:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:34:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.233.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab.
Entry for principal kudu/127.16.233.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab.
I20260810 22:34:43.822331 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:35971
--webserver_interface=127.16.233.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab
--principal=kudu/127.16.233.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:39053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.126:35971
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:34:43.945633 28739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:34:43.945935 28739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:34:43.946003 28739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:34:43.950222 28739 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:34:43.950297 28739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:34:43.950325 28739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:34:43.950343 28739 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:34:43.950359 28739 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:34:43.956383 28739 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:39053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.126:35971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab
--principal=kudu/127.16.233.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:35971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28739
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:34:43.957615 28739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:34:43.958729 28739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:34:43.967669 28748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:43.967687 28746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:43.967689 28744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:43.968065 28739 server_base.cc:1034] running on GCE node
I20260810 22:34:43.968664 28739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:34:43.969885 28739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:34:43.971086 28739 hybrid_clock.cc:648] HybridClock initialized: now 1786401283971043 us; error 59 us; skew 500 ppm
Aug 10 22:34:43 dist-test-slave-77v9 krb5kdc[28723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401283, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:34:43.974418 28739 init.cc:377] Logged in from keytab as kudu/127.16.233.126@KRBTEST.COM (short username kudu)
I20260810 22:34:43.975433 28739 webserver.cc:460] Webserver started at http://127.16.233.126:43593/ using document root <none> and password file <none>
I20260810 22:34:43.976027 28739 fs_manager.cc:359] Metadata directory not provided
I20260810 22:34:43.976123 28739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:34:43.976365 28739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:34:43.978579 28739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/minicluster-data/master-0/data/instance:
uuid: "b76a81e778bc4d7993d29ee46a4973ac"
format_stamp: "Formatted at 2026-08-10 22:34:43 on dist-test-slave-77v9"
server_key: "a8354341783a9a164b80133a46eedf58"
server_key_iv: "3cee4581a7f2ea9e3fe505ac1f3eee9f"
server_key_version: "encryptionkey@0"
I20260810 22:34:43.979184 28739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/minicluster-data/master-0/wal/instance:
uuid: "b76a81e778bc4d7993d29ee46a4973ac"
format_stamp: "Formatted at 2026-08-10 22:34:43 on dist-test-slave-77v9"
server_key: "a8354341783a9a164b80133a46eedf58"
server_key_iv: "3cee4581a7f2ea9e3fe505ac1f3eee9f"
server_key_version: "encryptionkey@0"
I20260810 22:34:43.983045 28739 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260810 22:34:43.985661 28756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:43.987088 28739 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 22:34:43.987236 28739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/minicluster-data/master-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/minicluster-data/master-0/wal
uuid: "b76a81e778bc4d7993d29ee46a4973ac"
format_stamp: "Formatted at 2026-08-10 22:34:43 on dist-test-slave-77v9"
server_key: "a8354341783a9a164b80133a46eedf58"
server_key_iv: "3cee4581a7f2ea9e3fe505ac1f3eee9f"
server_key_version: "encryptionkey@0"
I20260810 22:34:43.987361 28739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:34:44.003317 28739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:34:44.006512 28739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:34:44.006735 28739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:34:44.014457 28739 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.126:35971
I20260810 22:34:44.014781 28817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.126:35971 every 8 connection(s)
I20260810 22:34:44.015520 28739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
I20260810 22:34:44.018537 28818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:34:44.021039 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 28739
I20260810 22:34:44.021176 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/minicluster-data/master-0/wal/instance
I20260810 22:34:44.021466 17317 external_mini_cluster.cc:1428] Setting key 821f696b5210b03c61aa39106cc4f572
I20260810 22:34:44.026470 28818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b76a81e778bc4d7993d29ee46a4973ac: Bootstrap starting.
I20260810 22:34:44.029335 28818 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b76a81e778bc4d7993d29ee46a4973ac: Neither blocks nor log segments found. Creating new log.
I20260810 22:34:44.030349 28818 log.cc:824] T 00000000000000000000000000000000 P b76a81e778bc4d7993d29ee46a4973ac: Log is configured to *not* fsync() on all Append() calls
Aug 10 22:34:44 dist-test-slave-77v9 krb5kdc[28723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:34:44.032783 28818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b76a81e778bc4d7993d29ee46a4973ac: No bootstrap required, opened a new log
I20260810 22:34:44.037774 28818 raft_consensus.cc:348] T 00000000000000000000000000000000 P b76a81e778bc4d7993d29ee46a4973ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b76a81e778bc4d7993d29ee46a4973ac" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 35971 } }
I20260810 22:34:44.037989 28823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:44.022987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51956 (local address 127.16.233.126:35971)
0810 22:34:44.023473 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:44.023490 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:44.023527 (+ 37us) server_negotiation.cc:407] Connection header received
0810 22:34:44.024235 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:44.024284 (+ 49us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:44.024759 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:44.025178 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:44.026140 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:44.027309 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:44.028543 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:44.029138 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:44.029264 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:44.032156 (+ 2892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:44.032163 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:44.032180 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:44.032212 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:44.034430 (+ 2218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:44.034991 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:44.034998 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:44.035004 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:44.035161 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:44.035523 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:44.035529 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:44.035532 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:44.036109 (+ 577us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:44.036298 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:44.036600 (+ 302us) server_negotiation.cc:299] Negotiation successful
0810 22:34:44.036945 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":156,"threads_started":1}
I20260810 22:34:44.038080 28818 raft_consensus.cc:374] T 00000000000000000000000000000000 P b76a81e778bc4d7993d29ee46a4973ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:34:44.038127 28818 raft_consensus.cc:729] T 00000000000000000000000000000000 P b76a81e778bc4d7993d29ee46a4973ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b76a81e778bc4d7993d29ee46a4973ac, State: Initialized, Role: FOLLOWER
I20260810 22:34:44.038584 28818 consensus_queue.cc:260] T 00000000000000000000000000000000 P b76a81e778bc4d7993d29ee46a4973ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b76a81e778bc4d7993d29ee46a4973ac" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 35971 } }
I20260810 22:34:44.038757 28818 raft_consensus.cc:388] T 00000000000000000000000000000000 P b76a81e778bc4d7993d29ee46a4973ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:34:44.038811 28818 raft_consensus.cc:482] T 00000000000000000000000000000000 P b76a81e778bc4d7993d29ee46a4973ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:34:44.038905 28818 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b76a81e778bc4d7993d29ee46a4973ac [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:34:44.039906 28818 raft_consensus.cc:504] T 00000000000000000000000000000000 P b76a81e778bc4d7993d29ee46a4973ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b76a81e778bc4d7993d29ee46a4973ac" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 35971 } }
I20260810 22:34:44.040275 28818 leader_election.cc:302] T 00000000000000000000000000000000 P b76a81e778bc4d7993d29ee46a4973ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b76a81e778bc4d7993d29ee46a4973ac; no voters:
I20260810 22:34:44.040625 28818 leader_election.cc:288] T 00000000000000000000000000000000 P b76a81e778bc4d7993d29ee46a4973ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:34:44.040936 28825 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b76a81e778bc4d7993d29ee46a4973ac [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:34:44.041167 28825 raft_consensus.cc:686] T 00000000000000000000000000000000 P b76a81e778bc4d7993d29ee46a4973ac [term 1 LEADER]: Becoming Leader. State: Replica: b76a81e778bc4d7993d29ee46a4973ac, State: Running, Role: LEADER
I20260810 22:34:44.041484 28825 consensus_queue.cc:237] T 00000000000000000000000000000000 P b76a81e778bc4d7993d29ee46a4973ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b76a81e778bc4d7993d29ee46a4973ac" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 35971 } }
I20260810 22:34:44.041915 28818 sys_catalog.cc:564] T 00000000000000000000000000000000 P b76a81e778bc4d7993d29ee46a4973ac [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:34:44.043087 28827 sys_catalog.cc:455] T 00000000000000000000000000000000 P b76a81e778bc4d7993d29ee46a4973ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader b76a81e778bc4d7993d29ee46a4973ac. Latest consensus state: current_term: 1 leader_uuid: "b76a81e778bc4d7993d29ee46a4973ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b76a81e778bc4d7993d29ee46a4973ac" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 35971 } } }
I20260810 22:34:44.043233 28827 sys_catalog.cc:458] T 00000000000000000000000000000000 P b76a81e778bc4d7993d29ee46a4973ac [sys.catalog]: This master's current role is: LEADER
I20260810 22:34:44.043542 28826 sys_catalog.cc:455] T 00000000000000000000000000000000 P b76a81e778bc4d7993d29ee46a4973ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b76a81e778bc4d7993d29ee46a4973ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b76a81e778bc4d7993d29ee46a4973ac" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 35971 } } }
I20260810 22:34:44.043658 28826 sys_catalog.cc:458] T 00000000000000000000000000000000 P b76a81e778bc4d7993d29ee46a4973ac [sys.catalog]: This master's current role is: LEADER
I20260810 22:34:44.043788 28836 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:34:44.047063 28836 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:34:44.052459 28836 catalog_manager.cc:1347] Generated new cluster ID: 6ba7e404ac404840b69311fd301555b8
I20260810 22:34:44.052538 28836 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:34:44.076361 28836 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:34:44.077467 28836 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:34:44.084283 28836 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b76a81e778bc4d7993d29ee46a4973ac: Generated new TSK 0
I20260810 22:34:44.085105 28836 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.233.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab.
Entry for principal kudu/127.16.233.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab.
I20260810 22:34:44.125432 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.65:0
--local_ip_for_outbound_sockets=127.16.233.65
--webserver_interface=127.16.233.65
--webserver_port=0
--tserver_master_addrs=127.16.233.126:35971
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab
--principal=kudu/127.16.233.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:39053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:34:44.243034 28853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:34:44.243356 28853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:34:44.243436 28853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:34:44.247579 28853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:34:44.247648 28853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:34:44.247746 28853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.65
I20260810 22:34:44.253347 28853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:39053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab
--principal=kudu/127.16.233.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.233.65
--webserver_port=0
--tserver_master_addrs=127.16.233.126:35971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.65
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:34:44.254490 28853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:34:44.255599 28853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:34:44.265635 28862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:44.265617 28858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:44.265617 28859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:44.266505 28853 server_base.cc:1034] running on GCE node
I20260810 22:34:44.266995 28853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:34:44.267633 28853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:34:44.268870 28853 hybrid_clock.cc:648] HybridClock initialized: now 1786401284268798 us; error 33 us; skew 500 ppm
Aug 10 22:34:44 dist-test-slave-77v9 krb5kdc[28723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401284, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:34:44.272217 28853 init.cc:377] Logged in from keytab as kudu/127.16.233.65@KRBTEST.COM (short username kudu)
I20260810 22:34:44.273258 28853 webserver.cc:460] Webserver started at http://127.16.233.65:38939/ using document root <none> and password file <none>
I20260810 22:34:44.273870 28853 fs_manager.cc:359] Metadata directory not provided
I20260810 22:34:44.273942 28853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:34:44.274171 28853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:34:44.276381 28853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/minicluster-data/ts-0/data/instance:
uuid: "a39fef94b03c4f39ac51d914cdc0e74e"
format_stamp: "Formatted at 2026-08-10 22:34:44 on dist-test-slave-77v9"
server_key: "e25a1556e4e3f27ac14ec145df091114"
server_key_iv: "dd44c7c84ca386f15d56fcb545bd6205"
server_key_version: "encryptionkey@0"
I20260810 22:34:44.277045 28853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/minicluster-data/ts-0/wal/instance:
uuid: "a39fef94b03c4f39ac51d914cdc0e74e"
format_stamp: "Formatted at 2026-08-10 22:34:44 on dist-test-slave-77v9"
server_key: "e25a1556e4e3f27ac14ec145df091114"
server_key_iv: "dd44c7c84ca386f15d56fcb545bd6205"
server_key_version: "encryptionkey@0"
I20260810 22:34:44.281168 28853 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 22:34:44.283824 28871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:44.285197 28853 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:34:44.285351 28853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/minicluster-data/ts-0/wal
uuid: "a39fef94b03c4f39ac51d914cdc0e74e"
format_stamp: "Formatted at 2026-08-10 22:34:44 on dist-test-slave-77v9"
server_key: "e25a1556e4e3f27ac14ec145df091114"
server_key_iv: "dd44c7c84ca386f15d56fcb545bd6205"
server_key_version: "encryptionkey@0"
I20260810 22:34:44.285477 28853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:34:44.315478 28853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:34:44.319388 28853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:34:44.319650 28853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:34:44.320366 28853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:34:44.321292 28853 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:34:44.321364 28853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:44.321439 28853 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:34:44.321489 28853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:44.331212 28853 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.65:40699
I20260810 22:34:44.331244 29006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.65:40699 every 8 connection(s)
I20260810 22:34:44.332257 28853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
I20260810 22:34:44.333959 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 28853
I20260810 22:34:44.334121 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/minicluster-data/ts-0/wal/instance
I20260810 22:34:44.334444 17317 external_mini_cluster.cc:1428] Setting key c8703f7ccec9d850eb64eb6ff5233b3e
Aug 10 22:34:44 dist-test-slave-77v9 krb5kdc[28723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401284, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.65@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.233.66@KRBTEST.COM; defaulting to no policy
I20260810 22:34:44.349716 28823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:44.334340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:43211 (local address 127.16.233.126:35971)
0810 22:34:44.334496 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:44.334500 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:44.335570 (+ 1070us) server_negotiation.cc:407] Connection header received
0810 22:34:44.337782 (+ 2212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:44.337785 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:44.337875 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:44.337957 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:44.339538 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:44.340381 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:44.341966 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:44.342413 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:44.342575 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:44.345883 (+ 3308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:44.345890 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:44.345895 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:44.345933 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:44.347569 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:44.348154 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:44.348157 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:44.348159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:44.348240 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:44.348583 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:44.348585 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:44.348586 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:44.348773 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:44.348936 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:44.349474 (+ 538us) server_negotiation.cc:299] Negotiation successful
0810 22:34:44.349585 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 22:34:44.350719 29010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:44.334775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:35971 (local address 127.16.233.65:43211)
0810 22:34:44.335412 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:44.335461 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:44.337522 (+ 2061us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:44.338185 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:44.338198 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:44.338812 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:44.339367 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:44.339383 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:44.340536 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:44.340539 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:44.341830 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:44.341840 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:44.342662 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:44.342668 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:44.342879 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:44.343637 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:44.343653 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:44.345591 (+ 1938us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:34:44.347692 (+ 2101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:44.347698 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:44.347713 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:44.348015 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:44.348333 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:44.348335 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:44.348337 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:44.348489 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:44.348938 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:34:44.348943 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:34:44.349218 (+ 275us) client_negotiation.cc:769] Sending connection context
0810 22:34:44.349389 (+ 171us) client_negotiation.cc:240] Negotiation successful
0810 22:34:44.349695 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":149,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.66@KRBTEST.COM" created.
I20260810 22:34:44.352257 29008 heartbeater.cc:347] Connected to a master server at 127.16.233.126:35971
I20260810 22:34:44.352550 29008 heartbeater.cc:464] Registering TS with master...
I20260810 22:34:44.353235 29008 heartbeater.cc:511] Master 127.16.233.126:35971 requested a full tablet report, sending...
I20260810 22:34:44.354875 28779 ts_manager.cc:194] Registered new tserver with Master: a39fef94b03c4f39ac51d914cdc0e74e (127.16.233.65:40699)
I20260810 22:34:44.356148 28779 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.65@KRBTEST.COM'} at 127.16.233.65:43211
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab.
Entry for principal kudu/127.16.233.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab.
I20260810 22:34:44.373015 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.66:0
--local_ip_for_outbound_sockets=127.16.233.66
--webserver_interface=127.16.233.66
--webserver_port=0
--tserver_master_addrs=127.16.233.126:35971
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab
--principal=kudu/127.16.233.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:39053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:34:44.492178 29013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:34:44.492484 29013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:34:44.492530 29013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:34:44.496564 29013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:34:44.496639 29013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:34:44.496699 29013 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.66
I20260810 22:34:44.502329 29013 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:39053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab
--principal=kudu/127.16.233.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.233.66
--webserver_port=0
--tserver_master_addrs=127.16.233.126:35971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.66
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:34:44.503585 29013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:34:44.504691 29013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:34:44.514822 29024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:44.514822 29022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:44.514919 29020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:44.515254 29013 server_base.cc:1034] running on GCE node
I20260810 22:34:44.515704 29013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:34:44.516410 29013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:34:44.517589 29013 hybrid_clock.cc:648] HybridClock initialized: now 1786401284517552 us; error 48 us; skew 500 ppm
Aug 10 22:34:44 dist-test-slave-77v9 krb5kdc[28723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401284, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:34:44.528146 29013 init.cc:377] Logged in from keytab as kudu/127.16.233.66@KRBTEST.COM (short username kudu)
I20260810 22:34:44.529361 29013 webserver.cc:460] Webserver started at http://127.16.233.66:34755/ using document root <none> and password file <none>
I20260810 22:34:44.529928 29013 fs_manager.cc:359] Metadata directory not provided
I20260810 22:34:44.529994 29013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:34:44.530174 29013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:34:44.532466 29013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/minicluster-data/ts-1/data/instance:
uuid: "24ce87c8e8aa40f58a6e41495a814992"
format_stamp: "Formatted at 2026-08-10 22:34:44 on dist-test-slave-77v9"
server_key: "04574e03485c6740c2d1307d7c0cc415"
server_key_iv: "095c2a21636aab0b3048418858d732bb"
server_key_version: "encryptionkey@0"
I20260810 22:34:44.533128 29013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/minicluster-data/ts-1/wal/instance:
uuid: "24ce87c8e8aa40f58a6e41495a814992"
format_stamp: "Formatted at 2026-08-10 22:34:44 on dist-test-slave-77v9"
server_key: "04574e03485c6740c2d1307d7c0cc415"
server_key_iv: "095c2a21636aab0b3048418858d732bb"
server_key_version: "encryptionkey@0"
I20260810 22:34:44.537197 29013 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 22:34:44.539834 29032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:44.541289 29013 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:34:44.541399 29013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/minicluster-data/ts-1/wal
uuid: "24ce87c8e8aa40f58a6e41495a814992"
format_stamp: "Formatted at 2026-08-10 22:34:44 on dist-test-slave-77v9"
server_key: "04574e03485c6740c2d1307d7c0cc415"
server_key_iv: "095c2a21636aab0b3048418858d732bb"
server_key_version: "encryptionkey@0"
I20260810 22:34:44.541546 29013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:34:44.562036 29013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:34:44.565742 29013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:34:44.565968 29013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:34:44.566633 29013 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:34:44.567500 29013 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:34:44.567548 29013 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:44.567615 29013 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:34:44.567649 29013 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:44.577719 29013 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.66:39855
I20260810 22:34:44.577725 29159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.66:39855 every 8 connection(s)
I20260810 22:34:44.578734 29013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
I20260810 22:34:44.581460 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 29013
I20260810 22:34:44.581588 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/minicluster-data/ts-1/wal/instance
I20260810 22:34:44.581933 17317 external_mini_cluster.cc:1428] Setting key 2e7d642962764d6ae8fb1a575626ee3f
Aug 10 22:34:44 dist-test-slave-77v9 krb5kdc[28723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401284, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.66@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.233.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.67@KRBTEST.COM" created.
I20260810 22:34:44.596330 28823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:44.580937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:53481 (local address 127.16.233.126:35971)
0810 22:34:44.581094 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:44.581098 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:44.582324 (+ 1226us) server_negotiation.cc:407] Connection header received
0810 22:34:44.584097 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:44.584101 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:44.584195 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:44.584277 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:44.586105 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:44.587013 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:44.588523 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:44.588985 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:44.589118 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:44.592373 (+ 3255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:44.592377 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:44.592379 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:44.592411 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:44.593941 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:44.594586 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:44.594590 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:44.594591 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:44.594689 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:44.595107 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:44.595109 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:44.595111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:44.595310 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:44.595448 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:44.595994 (+ 546us) server_negotiation.cc:299] Negotiation successful
0810 22:34:44.596180 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 22:34:44.597354 29163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:44.581378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:35971 (local address 127.16.233.66:53481)
0810 22:34:44.582089 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:44.582152 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:44.583824 (+ 1672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:44.584529 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:44.584541 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:44.585134 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:44.585913 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:44.585938 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:44.587187 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:44.587190 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:44.588375 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:44.588386 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:44.589187 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:44.589193 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:44.589410 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:44.590120 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:44.590137 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:44.592165 (+ 2028us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:34:44.594065 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:44.594071 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:44.594085 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:44.594470 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:44.594863 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:44.594865 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:44.594866 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:44.595008 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:44.595453 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:34:44.595458 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:34:44.595719 (+ 261us) client_negotiation.cc:769] Sending connection context
0810 22:34:44.595907 (+ 188us) client_negotiation.cc:240] Negotiation successful
0810 22:34:44.596241 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":146,"threads_started":1}
I20260810 22:34:44.598718 29160 heartbeater.cc:347] Connected to a master server at 127.16.233.126:35971
I20260810 22:34:44.599040 29160 heartbeater.cc:464] Registering TS with master...
I20260810 22:34:44.599566 29160 heartbeater.cc:511] Master 127.16.233.126:35971 requested a full tablet report, sending...
I20260810 22:34:44.600724 28779 ts_manager.cc:194] Registered new tserver with Master: 24ce87c8e8aa40f58a6e41495a814992 (127.16.233.66:39855)
I20260810 22:34:44.601696 28779 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.66@KRBTEST.COM'} at 127.16.233.66:53481
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab.
Entry for principal kudu/127.16.233.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab.
I20260810 22:34:44.617730 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.67:0
--local_ip_for_outbound_sockets=127.16.233.67
--webserver_interface=127.16.233.67
--webserver_port=0
--tserver_master_addrs=127.16.233.126:35971
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab
--principal=kudu/127.16.233.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:39053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:34:44.736850 29166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:34:44.737149 29166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:34:44.737200 29166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:34:44.741796 29166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:34:44.741870 29166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:34:44.741953 29166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.67
I20260810 22:34:44.747843 29166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:39053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab
--principal=kudu/127.16.233.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.233.67
--webserver_port=0
--tserver_master_addrs=127.16.233.126:35971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.67
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:34:44.748983 29166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:34:44.750010 29166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:34:44.760913 29174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:34:44.760953 29171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:44.761042 29166 server_base.cc:1034] running on GCE node
W20260810 22:34:44.760977 29172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:34:44.761821 29166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:34:44.762642 29166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:34:44.763880 29166 hybrid_clock.cc:648] HybridClock initialized: now 1786401284763845 us; error 58 us; skew 500 ppm
Aug 10 22:34:44 dist-test-slave-77v9 krb5kdc[28723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401284, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:34:44.767515 29166 init.cc:377] Logged in from keytab as kudu/127.16.233.67@KRBTEST.COM (short username kudu)
I20260810 22:34:44.768545 29166 webserver.cc:460] Webserver started at http://127.16.233.67:42823/ using document root <none> and password file <none>
I20260810 22:34:44.769263 29166 fs_manager.cc:359] Metadata directory not provided
I20260810 22:34:44.769351 29166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:34:44.769568 29166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:34:44.772015 29166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/minicluster-data/ts-2/data/instance:
uuid: "cec87ed2f60848baa4eb8bb3a0f14343"
format_stamp: "Formatted at 2026-08-10 22:34:44 on dist-test-slave-77v9"
server_key: "c963eff17c7b28f982002822317d3f56"
server_key_iv: "95a39cef441a8bd45a840a5b7cfd3f64"
server_key_version: "encryptionkey@0"
I20260810 22:34:44.772704 29166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/minicluster-data/ts-2/wal/instance:
uuid: "cec87ed2f60848baa4eb8bb3a0f14343"
format_stamp: "Formatted at 2026-08-10 22:34:44 on dist-test-slave-77v9"
server_key: "c963eff17c7b28f982002822317d3f56"
server_key_iv: "95a39cef441a8bd45a840a5b7cfd3f64"
server_key_version: "encryptionkey@0"
I20260810 22:34:44.777654 29166 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260810 22:34:44.780375 29181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:44.782045 29166 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:34:44.782193 29166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/minicluster-data/ts-2/wal
uuid: "cec87ed2f60848baa4eb8bb3a0f14343"
format_stamp: "Formatted at 2026-08-10 22:34:44 on dist-test-slave-77v9"
server_key: "c963eff17c7b28f982002822317d3f56"
server_key_iv: "95a39cef441a8bd45a840a5b7cfd3f64"
server_key_version: "encryptionkey@0"
I20260810 22:34:44.782308 29166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:34:44.802953 29166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:34:44.806838 29166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:34:44.807022 29166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:34:44.807627 29166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:34:44.808521 29166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:34:44.808573 29166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:44.808625 29166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:34:44.808643 29166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:34:44.818797 29166 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.67:38685
I20260810 22:34:44.818831 29315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.67:38685 every 8 connection(s)
I20260810 22:34:44.819830 29166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
I20260810 22:34:44.826565 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 29166
I20260810 22:34:44.826748 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/minicluster-data/ts-2/wal/instance
I20260810 22:34:44.827116 17317 external_mini_cluster.cc:1428] Setting key e349c5db565102d3a82a02081b57157c
Aug 10 22:34:44 dist-test-slave-77v9 krb5kdc[28723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401284, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.67@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:34:44.835722 28823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:44.821954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:38381 (local address 127.16.233.126:35971)
0810 22:34:44.822153 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:44.822157 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:44.823133 (+ 976us) server_negotiation.cc:407] Connection header received
0810 22:34:44.824576 (+ 1443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:44.824579 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:44.824632 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:44.824712 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:44.826298 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:44.827030 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:44.828669 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:44.829028 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:44.829118 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:44.832032 (+ 2914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:44.832035 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:44.832038 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:44.832063 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:44.833670 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:44.834225 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:44.834228 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:44.834229 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:44.834312 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:44.834633 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:44.834635 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:44.834636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:44.834907 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:44.834984 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:44.835501 (+ 517us) server_negotiation.cc:299] Negotiation successful
0810 22:34:44.835599 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260810 22:34:44.836786 29318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:44.822386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:35971 (local address 127.16.233.67:38381)
0810 22:34:44.822953 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:44.823000 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:44.824335 (+ 1335us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:44.825084 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:44.825092 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:44.825520 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:34:44.826059 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:44.826076 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:44.827184 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:44.827189 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:44.828514 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:44.828527 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:44.829204 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:44.829208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:44.829426 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:44.830177 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:34:44.830195 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:34:44.831861 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:34:44.833786 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:44.833792 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:44.833809 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:44.834124 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:44.834400 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:34:44.834402 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:34:44.834404 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:34:44.834549 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:34:44.835000 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:34:44.835006 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:34:44.835263 (+ 257us) client_negotiation.cc:769] Sending connection context
0810 22:34:44.835461 (+ 198us) client_negotiation.cc:240] Negotiation successful
0810 22:34:44.835743 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":131,"threads_started":1}
I20260810 22:34:44.838164 29316 heartbeater.cc:347] Connected to a master server at 127.16.233.126:35971
I20260810 22:34:44.838495 29316 heartbeater.cc:464] Registering TS with master...
I20260810 22:34:44.839051 29316 heartbeater.cc:511] Master 127.16.233.126:35971 requested a full tablet report, sending...
I20260810 22:34:44.840263 28779 ts_manager.cc:194] Registered new tserver with Master: cec87ed2f60848baa4eb8bb3a0f14343 (127.16.233.67:38685)
I20260810 22:34:44.841154 28779 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.67@KRBTEST.COM'} at 127.16.233.67:38381
I20260810 22:34:44.842111 17317 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 22:34:44.853067 28823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:44.844579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51966 (local address 127.16.233.126:35971)
0810 22:34:44.844735 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:44.844740 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:44.845090 (+ 350us) server_negotiation.cc:407] Connection header received
0810 22:34:44.845181 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:44.845183 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:44.845224 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:44.845297 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:34:44.846276 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:44.847012 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:44.847896 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:44.848133 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:44.848222 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:44.849769 (+ 1547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:34:44.849772 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:34:44.849775 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:34:44.849846 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:34:44.851232 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:44.851704 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:44.851707 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:44.851708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:44.851788 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:34:44.852074 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:34:44.852076 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:34:44.852077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:34:44.852304 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:34:44.852413 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:44.852642 (+ 229us) server_negotiation.cc:299] Negotiation successful
0810 22:34:44.852744 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 22:34:44.856352 28778 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51966:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 22:34:44.859714 28778 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:34:44.877435 29330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:44.869900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.65:40699 (local address 127.0.0.1:55314)
0810 22:34:44.870907 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:44.870971 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:44.871129 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:44.871954 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:44.871961 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:44.871986 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:34:44.872296 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:44.872305 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:44.874126 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:44.874133 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:44.875461 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:44.875473 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:44.876683 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:44.876687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:44.876867 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:44.876939 (+ 72us) client_negotiation.cc:769] Sending connection context
0810 22:34:44.877077 (+ 138us) client_negotiation.cc:240] Negotiation successful
0810 22:34:44.877226 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":808,"mutex_wait_us":226,"thread_start_us":130,"threads_started":1}
I20260810 22:34:44.878123 29329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:44.870048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55314 (local address 127.16.233.65:40699)
0810 22:34:44.871205 (+ 1157us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:44.871211 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:44.871242 (+ 31us) server_negotiation.cc:407] Connection header received
0810 22:34:44.871336 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:44.871341 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:44.871574 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:44.871733 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:34:44.872467 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:44.873968 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:44.875628 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:44.876515 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:44.876657 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:44.876887 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:44.877837 (+ 950us) server_negotiation.cc:299] Negotiation successful
0810 22:34:44.877937 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1046,"thread_start_us":141,"threads_started":1}
I20260810 22:34:44.878414 29331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:44.870648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55522 (local address 127.16.233.66:39855)
0810 22:34:44.871053 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:44.871068 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:44.871450 (+ 382us) server_negotiation.cc:407] Connection header received
0810 22:34:44.871529 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:44.871534 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:44.871711 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:44.871847 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:34:44.872452 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:44.873839 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:44.876012 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:44.877031 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:44.877160 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:44.877310 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:44.878073 (+ 763us) server_negotiation.cc:299] Negotiation successful
0810 22:34:44.878191 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":196,"threads_started":1}
I20260810 22:34:44.877933 29328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:44.870103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.66:39855 (local address 127.0.0.1:55522)
0810 22:34:44.871209 (+ 1106us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:44.871227 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:44.871342 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:44.872029 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:44.872031 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:44.872064 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:34:44.872306 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:44.872313 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:44.874011 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:44.874018 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:44.875855 (+ 1837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:44.875880 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:44.877390 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:44.877395 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:44.877514 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:44.877545 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 22:34:44.877635 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 22:34:44.877741 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1041,"mutex_wait_us":200,"spinlock_wait_cycles":19328,"thread_start_us":127,"threads_started":1}
I20260810 22:34:44.878557 29332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:44.871200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.67:38685 (local address 127.0.0.1:34458)
0810 22:34:44.871611 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:44.871630 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:44.871757 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:44.872442 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:44.872445 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:44.872466 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:34:44.872579 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:44.872591 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:44.874205 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:44.874208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:44.875855 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:44.875868 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:44.878077 (+ 2209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:44.878082 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:44.878194 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:44.878232 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 22:34:44.878316 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 22:34:44.878390 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"mutex_wait_us":168,"thread_start_us":104,"threads_started":1}
I20260810 22:34:44.878937 29333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:44.871548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34458 (local address 127.16.233.67:38685)
0810 22:34:44.872006 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:44.872016 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:44.872037 (+ 21us) server_negotiation.cc:407] Connection header received
0810 22:34:44.872107 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:44.872111 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:44.872261 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:44.872444 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:34:44.872743 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:44.874031 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:44.876983 (+ 2952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:44.877916 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:44.878065 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:44.878227 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:44.878662 (+ 435us) server_negotiation.cc:299] Negotiation successful
0810 22:34:44.878770 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":120,"threads_started":1}
I20260810 22:34:44.881839 29240 tablet_service.cc:1467] Processing CreateTablet for tablet 08240c755fde4f23a29a97efa7fcb238 (DEFAULT_TABLE table=test-table [id=b69afa28b2b34453a12fee72cb0e3077]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:34:44.881846 28930 tablet_service.cc:1467] Processing CreateTablet for tablet 08240c755fde4f23a29a97efa7fcb238 (DEFAULT_TABLE table=test-table [id=b69afa28b2b34453a12fee72cb0e3077]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:34:44.881922 29084 tablet_service.cc:1467] Processing CreateTablet for tablet 08240c755fde4f23a29a97efa7fcb238 (DEFAULT_TABLE table=test-table [id=b69afa28b2b34453a12fee72cb0e3077]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:34:44.882942 29240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08240c755fde4f23a29a97efa7fcb238. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:34:44.883013 29084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08240c755fde4f23a29a97efa7fcb238. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:34:44.883013 28930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08240c755fde4f23a29a97efa7fcb238. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:34:44.890760 29335 tablet_bootstrap.cc:492] T 08240c755fde4f23a29a97efa7fcb238 P cec87ed2f60848baa4eb8bb3a0f14343: Bootstrap starting.
I20260810 22:34:44.892268 29336 tablet_bootstrap.cc:492] T 08240c755fde4f23a29a97efa7fcb238 P a39fef94b03c4f39ac51d914cdc0e74e: Bootstrap starting.
I20260810 22:34:44.892710 29337 tablet_bootstrap.cc:492] T 08240c755fde4f23a29a97efa7fcb238 P 24ce87c8e8aa40f58a6e41495a814992: Bootstrap starting.
I20260810 22:34:44.893989 29335 tablet_bootstrap.cc:654] T 08240c755fde4f23a29a97efa7fcb238 P cec87ed2f60848baa4eb8bb3a0f14343: Neither blocks nor log segments found. Creating new log.
I20260810 22:34:44.894766 29336 tablet_bootstrap.cc:654] T 08240c755fde4f23a29a97efa7fcb238 P a39fef94b03c4f39ac51d914cdc0e74e: Neither blocks nor log segments found. Creating new log.
I20260810 22:34:44.895174 29337 tablet_bootstrap.cc:654] T 08240c755fde4f23a29a97efa7fcb238 P 24ce87c8e8aa40f58a6e41495a814992: Neither blocks nor log segments found. Creating new log.
I20260810 22:34:44.895530 29335 log.cc:824] T 08240c755fde4f23a29a97efa7fcb238 P cec87ed2f60848baa4eb8bb3a0f14343: Log is configured to *not* fsync() on all Append() calls
I20260810 22:34:44.895574 29336 log.cc:824] T 08240c755fde4f23a29a97efa7fcb238 P a39fef94b03c4f39ac51d914cdc0e74e: Log is configured to *not* fsync() on all Append() calls
I20260810 22:34:44.895987 29337 log.cc:824] T 08240c755fde4f23a29a97efa7fcb238 P 24ce87c8e8aa40f58a6e41495a814992: Log is configured to *not* fsync() on all Append() calls
I20260810 22:34:44.898034 29335 tablet_bootstrap.cc:492] T 08240c755fde4f23a29a97efa7fcb238 P cec87ed2f60848baa4eb8bb3a0f14343: No bootstrap required, opened a new log
I20260810 22:34:44.898124 29337 tablet_bootstrap.cc:492] T 08240c755fde4f23a29a97efa7fcb238 P 24ce87c8e8aa40f58a6e41495a814992: No bootstrap required, opened a new log
I20260810 22:34:44.898124 29336 tablet_bootstrap.cc:492] T 08240c755fde4f23a29a97efa7fcb238 P a39fef94b03c4f39ac51d914cdc0e74e: No bootstrap required, opened a new log
I20260810 22:34:44.898290 29335 ts_tablet_manager.cc:1397] T 08240c755fde4f23a29a97efa7fcb238 P cec87ed2f60848baa4eb8bb3a0f14343: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260810 22:34:44.898334 29336 ts_tablet_manager.cc:1397] T 08240c755fde4f23a29a97efa7fcb238 P a39fef94b03c4f39ac51d914cdc0e74e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 22:34:44.898379 29337 ts_tablet_manager.cc:1397] T 08240c755fde4f23a29a97efa7fcb238 P 24ce87c8e8aa40f58a6e41495a814992: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 22:34:44.902124 29335 raft_consensus.cc:348] T 08240c755fde4f23a29a97efa7fcb238 P cec87ed2f60848baa4eb8bb3a0f14343 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39fef94b03c4f39ac51d914cdc0e74e" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 40699 } } peers { permanent_uuid: "24ce87c8e8aa40f58a6e41495a814992" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39855 } } peers { permanent_uuid: "cec87ed2f60848baa4eb8bb3a0f14343" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 38685 } }
I20260810 22:34:44.902436 29335 raft_consensus.cc:374] T 08240c755fde4f23a29a97efa7fcb238 P cec87ed2f60848baa4eb8bb3a0f14343 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:34:44.902518 29335 raft_consensus.cc:729] T 08240c755fde4f23a29a97efa7fcb238 P cec87ed2f60848baa4eb8bb3a0f14343 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec87ed2f60848baa4eb8bb3a0f14343, State: Initialized, Role: FOLLOWER
I20260810 22:34:44.902863 29337 raft_consensus.cc:348] T 08240c755fde4f23a29a97efa7fcb238 P 24ce87c8e8aa40f58a6e41495a814992 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39fef94b03c4f39ac51d914cdc0e74e" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 40699 } } peers { permanent_uuid: "24ce87c8e8aa40f58a6e41495a814992" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39855 } } peers { permanent_uuid: "cec87ed2f60848baa4eb8bb3a0f14343" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 38685 } }
I20260810 22:34:44.903035 29335 consensus_queue.cc:260] T 08240c755fde4f23a29a97efa7fcb238 P cec87ed2f60848baa4eb8bb3a0f14343 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39fef94b03c4f39ac51d914cdc0e74e" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 40699 } } peers { permanent_uuid: "24ce87c8e8aa40f58a6e41495a814992" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39855 } } peers { permanent_uuid: "cec87ed2f60848baa4eb8bb3a0f14343" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 38685 } }
I20260810 22:34:44.903172 29337 raft_consensus.cc:374] T 08240c755fde4f23a29a97efa7fcb238 P 24ce87c8e8aa40f58a6e41495a814992 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:34:44.903270 29337 raft_consensus.cc:729] T 08240c755fde4f23a29a97efa7fcb238 P 24ce87c8e8aa40f58a6e41495a814992 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24ce87c8e8aa40f58a6e41495a814992, State: Initialized, Role: FOLLOWER
I20260810 22:34:44.903404 29336 raft_consensus.cc:348] T 08240c755fde4f23a29a97efa7fcb238 P a39fef94b03c4f39ac51d914cdc0e74e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39fef94b03c4f39ac51d914cdc0e74e" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 40699 } } peers { permanent_uuid: "24ce87c8e8aa40f58a6e41495a814992" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39855 } } peers { permanent_uuid: "cec87ed2f60848baa4eb8bb3a0f14343" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 38685 } }
I20260810 22:34:44.903734 29336 raft_consensus.cc:374] T 08240c755fde4f23a29a97efa7fcb238 P a39fef94b03c4f39ac51d914cdc0e74e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:34:44.903782 29316 heartbeater.cc:503] Master 127.16.233.126:35971 was elected leader, sending a full tablet report...
I20260810 22:34:44.903805 29337 consensus_queue.cc:260] T 08240c755fde4f23a29a97efa7fcb238 P 24ce87c8e8aa40f58a6e41495a814992 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39fef94b03c4f39ac51d914cdc0e74e" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 40699 } } peers { permanent_uuid: "24ce87c8e8aa40f58a6e41495a814992" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39855 } } peers { permanent_uuid: "cec87ed2f60848baa4eb8bb3a0f14343" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 38685 } }
I20260810 22:34:44.903903 29336 raft_consensus.cc:729] T 08240c755fde4f23a29a97efa7fcb238 P a39fef94b03c4f39ac51d914cdc0e74e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a39fef94b03c4f39ac51d914cdc0e74e, State: Initialized, Role: FOLLOWER
I20260810 22:34:44.904217 29335 ts_tablet_manager.cc:1428] T 08240c755fde4f23a29a97efa7fcb238 P cec87ed2f60848baa4eb8bb3a0f14343: Time spent starting tablet: real 0.006s user 0.001s sys 0.006s
I20260810 22:34:44.904397 29336 consensus_queue.cc:260] T 08240c755fde4f23a29a97efa7fcb238 P a39fef94b03c4f39ac51d914cdc0e74e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39fef94b03c4f39ac51d914cdc0e74e" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 40699 } } peers { permanent_uuid: "24ce87c8e8aa40f58a6e41495a814992" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39855 } } peers { permanent_uuid: "cec87ed2f60848baa4eb8bb3a0f14343" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 38685 } }
I20260810 22:34:44.905169 29008 heartbeater.cc:503] Master 127.16.233.126:35971 was elected leader, sending a full tablet report...
I20260810 22:34:44.905413 29336 ts_tablet_manager.cc:1428] T 08240c755fde4f23a29a97efa7fcb238 P a39fef94b03c4f39ac51d914cdc0e74e: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260810 22:34:44.907043 29337 ts_tablet_manager.cc:1428] T 08240c755fde4f23a29a97efa7fcb238 P 24ce87c8e8aa40f58a6e41495a814992: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260810 22:34:44.907402 29160 heartbeater.cc:503] Master 127.16.233.126:35971 was elected leader, sending a full tablet report...
W20260810 22:34:45.071038 29317 tablet.cc:2369] T 08240c755fde4f23a29a97efa7fcb238 P cec87ed2f60848baa4eb8bb3a0f14343: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 22:34:45.080086 29162 tablet.cc:2369] T 08240c755fde4f23a29a97efa7fcb238 P 24ce87c8e8aa40f58a6e41495a814992: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 22:34:45.083811 29009 tablet.cc:2369] T 08240c755fde4f23a29a97efa7fcb238 P a39fef94b03c4f39ac51d914cdc0e74e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:34:45.178401 29346 raft_consensus.cc:482] T 08240c755fde4f23a29a97efa7fcb238 P a39fef94b03c4f39ac51d914cdc0e74e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:34:45.178710 29346 raft_consensus.cc:504] T 08240c755fde4f23a29a97efa7fcb238 P a39fef94b03c4f39ac51d914cdc0e74e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39fef94b03c4f39ac51d914cdc0e74e" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 40699 } } peers { permanent_uuid: "24ce87c8e8aa40f58a6e41495a814992" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39855 } } peers { permanent_uuid: "cec87ed2f60848baa4eb8bb3a0f14343" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 38685 } }
I20260810 22:34:45.179973 29346 leader_election.cc:288] T 08240c755fde4f23a29a97efa7fcb238 P a39fef94b03c4f39ac51d914cdc0e74e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 24ce87c8e8aa40f58a6e41495a814992 (127.16.233.66:39855), cec87ed2f60848baa4eb8bb3a0f14343 (127.16.233.67:38685)
I20260810 22:34:45.184423 29333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:45.180435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:35711 (local address 127.16.233.67:38685)
0810 22:34:45.180571 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:45.180577 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:45.181045 (+ 468us) server_negotiation.cc:407] Connection header received
0810 22:34:45.181201 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:45.181204 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:45.181260 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:45.181352 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:34:45.181636 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:45.182131 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:45.183347 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:45.183757 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:45.183895 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:45.183948 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:45.184200 (+ 252us) server_negotiation.cc:299] Negotiation successful
0810 22:34:45.184291 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 22:34:45.184424 29331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:45.180434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:56047 (local address 127.16.233.66:39855)
0810 22:34:45.180571 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:45.180576 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:45.180871 (+ 295us) server_negotiation.cc:407] Connection header received
0810 22:34:45.181071 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:45.181073 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:45.181118 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:45.181201 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:34:45.181525 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:45.182127 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:45.183348 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:45.183750 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:45.183881 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:45.183938 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:45.184185 (+ 247us) server_negotiation.cc:299] Negotiation successful
0810 22:34:45.184273 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 22:34:45.184459 29349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:45.180319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.66:39855 (local address 127.16.233.65:56047)
0810 22:34:45.180745 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:45.180759 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:45.180944 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:45.181226 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:45.181229 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:45.181267 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:34:45.181405 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:45.181411 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:45.182263 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:45.182267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:45.183211 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:45.183220 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:45.183882 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:45.183886 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:45.184042 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:45.184073 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 22:34:45.184165 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 22:34:45.184299 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"mutex_wait_us":95,"thread_start_us":135,"threads_started":1}
I20260810 22:34:45.184463 29347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:45.180318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.67:38685 (local address 127.16.233.65:35711)
0810 22:34:45.180927 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:45.180940 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 22:34:45.181082 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:34:45.181375 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:34:45.181378 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:34:45.181395 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:34:45.181519 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:45.181524 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:45.182249 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:45.182254 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:45.183203 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:34:45.183213 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:34:45.183893 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:34:45.183897 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:34:45.184054 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:45.184088 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 22:34:45.184181 (+ 93us) client_negotiation.cc:240] Negotiation successful
0810 22:34:45.184291 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"mutex_wait_us":134,"thread_start_us":96,"threads_started":1}
I20260810 22:34:45.185294 29106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08240c755fde4f23a29a97efa7fcb238" candidate_uuid: "a39fef94b03c4f39ac51d914cdc0e74e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24ce87c8e8aa40f58a6e41495a814992" is_pre_election: true
I20260810 22:34:45.185433 29264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08240c755fde4f23a29a97efa7fcb238" candidate_uuid: "a39fef94b03c4f39ac51d914cdc0e74e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec87ed2f60848baa4eb8bb3a0f14343" is_pre_election: true
I20260810 22:34:45.185688 29264 raft_consensus.cc:2454] T 08240c755fde4f23a29a97efa7fcb238 P cec87ed2f60848baa4eb8bb3a0f14343 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a39fef94b03c4f39ac51d914cdc0e74e in term 0.
I20260810 22:34:45.185688 29106 raft_consensus.cc:2454] T 08240c755fde4f23a29a97efa7fcb238 P 24ce87c8e8aa40f58a6e41495a814992 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a39fef94b03c4f39ac51d914cdc0e74e in term 0.
I20260810 22:34:45.186368 28876 leader_election.cc:302] T 08240c755fde4f23a29a97efa7fcb238 P a39fef94b03c4f39ac51d914cdc0e74e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 24ce87c8e8aa40f58a6e41495a814992, a39fef94b03c4f39ac51d914cdc0e74e, cec87ed2f60848baa4eb8bb3a0f14343; no voters:
I20260810 22:34:45.186748 29346 raft_consensus.cc:2781] T 08240c755fde4f23a29a97efa7fcb238 P a39fef94b03c4f39ac51d914cdc0e74e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:34:45.186909 29346 raft_consensus.cc:482] T 08240c755fde4f23a29a97efa7fcb238 P a39fef94b03c4f39ac51d914cdc0e74e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:34:45.186983 29346 raft_consensus.cc:3037] T 08240c755fde4f23a29a97efa7fcb238 P a39fef94b03c4f39ac51d914cdc0e74e [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:34:45.188099 29346 raft_consensus.cc:504] T 08240c755fde4f23a29a97efa7fcb238 P a39fef94b03c4f39ac51d914cdc0e74e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39fef94b03c4f39ac51d914cdc0e74e" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 40699 } } peers { permanent_uuid: "24ce87c8e8aa40f58a6e41495a814992" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39855 } } peers { permanent_uuid: "cec87ed2f60848baa4eb8bb3a0f14343" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 38685 } }
I20260810 22:34:45.188519 29346 leader_election.cc:288] T 08240c755fde4f23a29a97efa7fcb238 P a39fef94b03c4f39ac51d914cdc0e74e [CANDIDATE]: Term 1 election: Requested vote from peers 24ce87c8e8aa40f58a6e41495a814992 (127.16.233.66:39855), cec87ed2f60848baa4eb8bb3a0f14343 (127.16.233.67:38685)
I20260810 22:34:45.189029 29264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08240c755fde4f23a29a97efa7fcb238" candidate_uuid: "a39fef94b03c4f39ac51d914cdc0e74e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec87ed2f60848baa4eb8bb3a0f14343"
I20260810 22:34:45.189033 29106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08240c755fde4f23a29a97efa7fcb238" candidate_uuid: "a39fef94b03c4f39ac51d914cdc0e74e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24ce87c8e8aa40f58a6e41495a814992"
I20260810 22:34:45.189184 29106 raft_consensus.cc:3037] T 08240c755fde4f23a29a97efa7fcb238 P 24ce87c8e8aa40f58a6e41495a814992 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:34:45.189184 29264 raft_consensus.cc:3037] T 08240c755fde4f23a29a97efa7fcb238 P cec87ed2f60848baa4eb8bb3a0f14343 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:34:45.190615 29106 raft_consensus.cc:2454] T 08240c755fde4f23a29a97efa7fcb238 P 24ce87c8e8aa40f58a6e41495a814992 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a39fef94b03c4f39ac51d914cdc0e74e in term 1.
I20260810 22:34:45.190614 29264 raft_consensus.cc:2454] T 08240c755fde4f23a29a97efa7fcb238 P cec87ed2f60848baa4eb8bb3a0f14343 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a39fef94b03c4f39ac51d914cdc0e74e in term 1.
I20260810 22:34:45.191018 28877 leader_election.cc:302] T 08240c755fde4f23a29a97efa7fcb238 P a39fef94b03c4f39ac51d914cdc0e74e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a39fef94b03c4f39ac51d914cdc0e74e, cec87ed2f60848baa4eb8bb3a0f14343; no voters:
I20260810 22:34:45.191294 29346 raft_consensus.cc:2781] T 08240c755fde4f23a29a97efa7fcb238 P a39fef94b03c4f39ac51d914cdc0e74e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:34:45.191560 29346 raft_consensus.cc:686] T 08240c755fde4f23a29a97efa7fcb238 P a39fef94b03c4f39ac51d914cdc0e74e [term 1 LEADER]: Becoming Leader. State: Replica: a39fef94b03c4f39ac51d914cdc0e74e, State: Running, Role: LEADER
I20260810 22:34:45.191982 29346 consensus_queue.cc:237] T 08240c755fde4f23a29a97efa7fcb238 P a39fef94b03c4f39ac51d914cdc0e74e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39fef94b03c4f39ac51d914cdc0e74e" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 40699 } } peers { permanent_uuid: "24ce87c8e8aa40f58a6e41495a814992" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39855 } } peers { permanent_uuid: "cec87ed2f60848baa4eb8bb3a0f14343" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 38685 } }
I20260810 22:34:45.195497 28777 catalog_manager.cc:5495] T 08240c755fde4f23a29a97efa7fcb238 P a39fef94b03c4f39ac51d914cdc0e74e reported cstate change: term changed from 0 to 1, leader changed from <none> to a39fef94b03c4f39ac51d914cdc0e74e (127.16.233.65). New cstate: current_term: 1 leader_uuid: "a39fef94b03c4f39ac51d914cdc0e74e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39fef94b03c4f39ac51d914cdc0e74e" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 40699 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24ce87c8e8aa40f58a6e41495a814992" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec87ed2f60848baa4eb8bb3a0f14343" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 38685 } health_report { overall_health: UNKNOWN } } }
I20260810 22:34:45.243891 29329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:45.239431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55316 (local address 127.16.233.65:40699)
0810 22:34:45.239593 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 22:34:45.239598 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:34:45.239614 (+ 16us) server_negotiation.cc:407] Connection header received
0810 22:34:45.239755 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:34:45.239758 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:34:45.239829 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:34:45.239965 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:34:45.240227 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:45.240742 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:45.241792 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:34:45.242019 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:34:45.242136 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:34:45.242578 (+ 442us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:34:45.243196 (+ 618us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:34:45.243406 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0810 22:34:45.243493 (+ 87us) server_negotiation.cc:299] Negotiation successful
0810 22:34:45.243573 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 22:34:45.251016 29264 raft_consensus.cc:1261] T 08240c755fde4f23a29a97efa7fcb238 P cec87ed2f60848baa4eb8bb3a0f14343 [term 1 FOLLOWER]: Refusing update from remote peer a39fef94b03c4f39ac51d914cdc0e74e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:34:45.251016 29106 raft_consensus.cc:1261] T 08240c755fde4f23a29a97efa7fcb238 P 24ce87c8e8aa40f58a6e41495a814992 [term 1 FOLLOWER]: Refusing update from remote peer a39fef94b03c4f39ac51d914cdc0e74e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:34:45.251834 29346 consensus_queue.cc:1035] T 08240c755fde4f23a29a97efa7fcb238 P a39fef94b03c4f39ac51d914cdc0e74e [LEADER]: Connected to new peer: Peer: permanent_uuid: "24ce87c8e8aa40f58a6e41495a814992" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 39855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:34:45.252156 29350 consensus_queue.cc:1035] T 08240c755fde4f23a29a97efa7fcb238 P a39fef94b03c4f39ac51d914cdc0e74e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec87ed2f60848baa4eb8bb3a0f14343" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 38685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:34:45.261605 29355 mvcc.cc:205] Tried to move back new op lower bound from 7317099664380170240 to 7317099664149848064. Current Snapshot: MvccSnapshot[applied={T|T < 7317099664380170240}]
I20260810 22:34:45.262079 29354 mvcc.cc:205] Tried to move back new op lower bound from 7317099664380170240 to 7317099664149848064. Current Snapshot: MvccSnapshot[applied={T|T < 7317099664380170240}]
I20260810 22:34:45.270483 28779 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51966:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:34:45.270701 28779 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51966:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:34:45.273733 28779 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b76a81e778bc4d7993d29ee46a4973ac: Sending DeleteTablet for 3 replicas of tablet 08240c755fde4f23a29a97efa7fcb238
I20260810 22:34:45.274884 29084 tablet_service.cc:1514] Processing DeleteTablet for tablet 08240c755fde4f23a29a97efa7fcb238 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:34:45 UTC) from {username='kudu', principal='kudu/127.16.233.126@KRBTEST.COM'} at 127.0.0.1:55522
I20260810 22:34:45.274900 28930 tablet_service.cc:1514] Processing DeleteTablet for tablet 08240c755fde4f23a29a97efa7fcb238 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:34:45 UTC) from {username='kudu', principal='kudu/127.16.233.126@KRBTEST.COM'} at 127.0.0.1:55314
I20260810 22:34:45.275518 29364 tablet_replica.cc:330] T 08240c755fde4f23a29a97efa7fcb238 P a39fef94b03c4f39ac51d914cdc0e74e: stopping tablet replica
I20260810 22:34:45.275786 29240 tablet_service.cc:1514] Processing DeleteTablet for tablet 08240c755fde4f23a29a97efa7fcb238 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:34:45 UTC) from {username='kudu', principal='kudu/127.16.233.126@KRBTEST.COM'} at 127.0.0.1:34458
I20260810 22:34:45.276016 29364 raft_consensus.cc:2229] T 08240c755fde4f23a29a97efa7fcb238 P a39fef94b03c4f39ac51d914cdc0e74e [term 1 LEADER]: Raft consensus shutting down.
I20260810 22:34:45.276332 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 28853
I20260810 22:34:45.276536 29364 raft_consensus.cc:2258] T 08240c755fde4f23a29a97efa7fcb238 P a39fef94b03c4f39ac51d914cdc0e74e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:34:45.277647 29365 tablet_replica.cc:330] T 08240c755fde4f23a29a97efa7fcb238 P 24ce87c8e8aa40f58a6e41495a814992: stopping tablet replica
I20260810 22:34:45.278062 29365 raft_consensus.cc:2229] T 08240c755fde4f23a29a97efa7fcb238 P 24ce87c8e8aa40f58a6e41495a814992 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:34:45.278416 29365 raft_consensus.cc:2258] T 08240c755fde4f23a29a97efa7fcb238 P 24ce87c8e8aa40f58a6e41495a814992 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:34:45.281574 29366 tablet_replica.cc:330] T 08240c755fde4f23a29a97efa7fcb238 P cec87ed2f60848baa4eb8bb3a0f14343: stopping tablet replica
I20260810 22:34:45.282037 29366 raft_consensus.cc:2229] T 08240c755fde4f23a29a97efa7fcb238 P cec87ed2f60848baa4eb8bb3a0f14343 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:34:45.282407 29366 raft_consensus.cc:2258] T 08240c755fde4f23a29a97efa7fcb238 P cec87ed2f60848baa4eb8bb3a0f14343 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:34:45.282462 29365 ts_tablet_manager.cc:1905] T 08240c755fde4f23a29a97efa7fcb238 P 24ce87c8e8aa40f58a6e41495a814992: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:34:45.283807 29366 ts_tablet_manager.cc:1905] T 08240c755fde4f23a29a97efa7fcb238 P cec87ed2f60848baa4eb8bb3a0f14343: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:34:45.285516 29365 ts_tablet_manager.cc:1918] T 08240c755fde4f23a29a97efa7fcb238 P 24ce87c8e8aa40f58a6e41495a814992: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:34:45.285638 29365 log.cc:1196] T 08240c755fde4f23a29a97efa7fcb238 P 24ce87c8e8aa40f58a6e41495a814992: Deleting WAL directory at /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/minicluster-data/ts-1/wal/wals/08240c755fde4f23a29a97efa7fcb238
I20260810 22:34:45.286049 29365 ts_tablet_manager.cc:1939] T 08240c755fde4f23a29a97efa7fcb238 P 24ce87c8e8aa40f58a6e41495a814992: Deleting consensus metadata
I20260810 22:34:45.286511 29366 ts_tablet_manager.cc:1918] T 08240c755fde4f23a29a97efa7fcb238 P cec87ed2f60848baa4eb8bb3a0f14343: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:34:45.286653 29366 log.cc:1196] T 08240c755fde4f23a29a97efa7fcb238 P cec87ed2f60848baa4eb8bb3a0f14343: Deleting WAL directory at /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786401215402979-17317-0/minicluster-data/ts-2/wal/wals/08240c755fde4f23a29a97efa7fcb238
W20260810 22:34:45.286679 28757 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.233.65:40699 (error 108)
I20260810 22:34:45.287047 29366 ts_tablet_manager.cc:1939] T 08240c755fde4f23a29a97efa7fcb238 P cec87ed2f60848baa4eb8bb3a0f14343: Deleting consensus metadata
I20260810 22:34:45.287751 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 29013
I20260810 22:34:45.287950 28764 catalog_manager.cc:4841] TS 24ce87c8e8aa40f58a6e41495a814992 (127.16.233.66:39855): tablet 08240c755fde4f23a29a97efa7fcb238 (table test-table [id=b69afa28b2b34453a12fee72cb0e3077]) successfully deleted
I20260810 22:34:45.288621 29328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:34:45.287616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.65:40699 (local address 127.0.0.1:55326)
0810 22:34:45.288388 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0810 22:34:45.288491 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.65:40699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":637}
W20260810 22:34:45.288839 28757 catalog_manager.cc:4584] TS a39fef94b03c4f39ac51d914cdc0e74e (127.16.233.65:40699): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 08240c755fde4f23a29a97efa7fcb238: Network error: Client connection negotiation failed: client connection to 127.16.233.65:40699: connect: Connection refused (error 111)
I20260810 22:34:45.291347 28765 catalog_manager.cc:4841] TS cec87ed2f60848baa4eb8bb3a0f14343 (127.16.233.67:38685): tablet 08240c755fde4f23a29a97efa7fcb238 (table test-table [id=b69afa28b2b34453a12fee72cb0e3077]) successfully deleted
I20260810 22:34:45.298218 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 29166
I20260810 22:34:45.307497 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 28739
2026-08-10T22:34:45Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2799 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:34:45 dist-test-slave-77v9 krb5kdc[29371](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:34:45 dist-test-slave-77v9 krb5kdc[29371](info): set up 2 sockets
Aug 10 22:34:45 dist-test-slave-77v9 krb5kdc[29371](info): commencing operation
krb5kdc: starting...
W20260810 22:34:46.497995 17317 mini_kdc.cc:121] Time spent starting KDC: real 1.163s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Aug 10 22:34:46 dist-test-slave-77v9 krb5kdc[29371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:34:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:34:46Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.16.233.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.16.233.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.16.233.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/krb5kdc/rangeradmin_127.16.233.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.16.233.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/krb5kdc/rangeradmin_127.16.233.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.16.233.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.16.233.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.16.233.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/krb5kdc/rangerlookup_127.16.233.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.16.233.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/krb5kdc/rangerlookup_127.16.233.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.233.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.16.233.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.233.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/krb5kdc/HTTP_127.16.233.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.233.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/krb5kdc/HTTP_127.16.233.84@KRBTEST.COM.keytab.
I20260810 22:34:46.717293 17317 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskfcURSo/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/postgres -l logfile start
2026-08-10 22:34:49.650 UTC [29405] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-10 22:34:49.650 UTC [29405] LOG: listening on IPv4 address "127.16.233.84", port 57811
2026-08-10 22:34:49.655 UTC [29405] LOG: listening on Unix socket "/tmp/.s.PGSQL.57811"
2026-08-10 22:34:49.678 UTC [29408] LOG: database system was shut down at 2026-08-10 22:34:47 UTC
2026-08-10 22:34:49.682 UTC [29405] LOG: database system is ready to accept connections
I20260810 22:34:50.723835 17317 mini_postgres.cc:96] Postgres bound to 57811
2026-08-10 22:34:50.731 UTC [29418] FATAL: database "slave" does not exist
127.16.233.84:57811 - accepting connections
I20260810 22:34:50.732924 17317 mini_ranger.cc:161] Starting Ranger...
I20260810 22:34:50.751482 17317 mini_ranger.cc:84] Created miniranger Postgres user
I20260810 22:34:51.417850 17317 mini_ranger.cc:87] Created ranger Postgres database
I20260810 22:34:51.417982 17317 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/ranger-admin
2026-08-10 22:34:51,738 [I] DB FLAVOR :POSTGRES
2026-08-10 22:34:51,739 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 22:34:51,739 [I] Checking connection..
2026-08-10 22:34:51,739 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfcURSo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfcURSo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.84:57811/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-10 22:34:52,013 [I] Checking connection passed.
2026-08-10 22:34:52,013 [I] --------- Verifying version history table ---------
2026-08-10 22:34:52,013 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfcURSo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfcURSo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.84:57811/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 22:34:52,300 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 22:34:52,300 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfcURSo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfcURSo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.84:57811/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 22:34:52,594 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 22:34:52,594 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-10 22:34:52,595 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfcURSo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfcURSo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.84:57811/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskfcURSo/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-10 22:34:52.851 UTC [29490] WARNING: there is no transaction in progress
2026-08-10 22:34:52,859 [I] create_dbversion_catalog.sql file imported successfully
2026-08-10 22:34:52,859 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfcURSo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfcURSo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.84:57811/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 22:34:53,123 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-10 22:34:53,123 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 22:34:53,123 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfcURSo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfcURSo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.84:57811/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-10 22:34:53,441 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfcURSo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfcURSo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.84:57811/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-10 22:34:53,730 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfcURSo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfcURSo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.84:57811/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-77v9.c.gcp-upstream.internal','N') ;"
2026-08-10 22:34:54,006 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfcURSo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfcURSo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.84:57811/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 22:34:54,262 [I] Table x_portal_user does not exist in database ranger
2026-08-10 22:34:54,263 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfcURSo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfcURSo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.84:57811/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 22:34:54,518 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-10 22:34:54,518 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfcURSo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfcURSo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.84:57811/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 22:34:54,780 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfcURSo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfcURSo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.84:57811/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 22:34:55,055 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-10 22:34:55,055 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfcURSo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfcURSo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.84:57811/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskfcURSo/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-10 22:34:56.163 UTC [29629] WARNING: there is no transaction in progress
2026-08-10 22:34:56.182 UTC [29629] WARNING: there is no transaction in progress
2026-08-10 22:34:56.201 UTC [29629] WARNING: there is no transaction in progress
2026-08-10 22:34:56.219 UTC [29629] WARNING: there is no transaction in progress
2026-08-10 22:34:56.239 UTC [29629] WARNING: there is no transaction in progress
2026-08-10 22:34:56.262 UTC [29629] WARNING: there is no transaction in progress
2026-08-10 22:34:56.331 UTC [29629] WARNING: there is no transaction in progress
2026-08-10 22:34:56.339 UTC [29629] WARNING: there is no transaction in progress
2026-08-10 22:34:56.355 UTC [29629] WARNING: there is no transaction in progress
2026-08-10 22:34:56.366 UTC [29629] WARNING: there is no transaction in progress
2026-08-10 22:34:56.377 UTC [29629] WARNING: there is no transaction in progress
2026-08-10 22:34:56.393 UTC [29629] WARNING: there is no transaction in progress
2026-08-10 22:34:56.405 UTC [29629] WARNING: there is no transaction in progress
2026-08-10 22:34:56.413 UTC [29629] WARNING: there is no transaction in progress
2026-08-10 22:34:56.422 UTC [29629] WARNING: there is no transaction in progress
2026-08-10 22:34:56.430 UTC [29629] WARNING: there is no transaction in progress
2026-08-10 22:34:57,259 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-10 22:34:57,259 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfcURSo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfcURSo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.84:57811/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-10 22:34:57,523 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-10 22:34:57,523 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfcURSo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfcURSo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.84:57811/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 22:34:57,804 [I] Table x_portal_user already exists in database 'ranger'
2026-08-10 22:34:57,804 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfcURSo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfcURSo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.84:57811/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 22:34:58,092 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-10 22:34:58,092 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfcURSo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfcURSo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.84:57811/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 22:34:58,356 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfcURSo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfcURSo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.84:57811/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 22:34:58,646 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfcURSo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfcURSo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.84:57811/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-77v9.c.gcp-upstream.internal';"
2026-08-10 22:34:58,920 [I] CORE_DB_SCHEMA import status has been updated
2026-08-10 22:34:58,921 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfcURSo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfcURSo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.84:57811/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-10 22:34:59,174 [I] DB_PATCHES have already been applied
I20260810 22:34:59.180987 17317 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/ranger-admin:/tmp/dist-test-taskfcURSo/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskfcURSo/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskfcURSo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfcURSo/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260810 22:34:59.181092 17317 mini_ranger.cc:193] Using host: 127.16.233.84
I20260810 22:34:59.186542 17317 mini_ranger.cc:227] Ranger admin URL: http://127.16.233.84:57707
Aug 10, 2026 10:34:59 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 10:34:59 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 10:34:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/ranger-admin/ews/webapp
Aug 10, 2026 10:34:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 10, 2026 10:34:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 10, 2026 10:35:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/ranger-admin/ews/webapp].
Aug 10, 2026 10:35:00 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260810 22:35:00.439393 17317 mini_ranger.cc:160] Time spent starting Ranger: real 9.707s user 0.000s sys 0.016s
Aug 10, 2026 10:35:23 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 10, 2026 10:35:23 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 10, 2026 10:35:23 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 10, 2026 10:35:23 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 10, 2026 10:35:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 10, 2026 10:35:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 10, 2026 10:35:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 10, 2026 10:35:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 10, 2026 10:35:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 10, 2026 10:35:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 10, 2026 10:35:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 10, 2026 10:35:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 10, 2026 10:35:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 10, 2026 10:35:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 10, 2026 10:35:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 10, 2026 10:35:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 10, 2026 10:35:23 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 10, 2026 10:35:24 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260810 22:35:25.276774 17317 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.16.233.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.16.233.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.16.233.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/krb5kdc/rangerkms_127.16.233.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.16.233.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/krb5kdc/rangerkms_127.16.233.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.233.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.16.233.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.233.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/krb5kdc/HTTP_127.16.233.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.233.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/krb5kdc/HTTP_127.16.233.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 10 22:35:25 dist-test-slave-77v9 krb5kdc[29371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401325, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260810 22:35:25.384194 17317 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260810 22:35:25.397089 17317 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260810 22:35:26.435088 17317 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260810 22:35:26.435209 17317 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/ranger-kms
I20260810 22:35:26.435328 17317 mini_ranger_kms.cc:227] Using postgres at 127.16.233.84:57811
2026-08-10 22:35:27,181 [I] DB FLAVOR :POSTGRES
2026-08-10 22:35:27,181 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 22:35:27,181 [I] Checking connection
2026-08-10 22:35:27,182 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfcURSo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.84:57811/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-10 22:35:27,570 [I] connection success
2026-08-10 22:35:27,570 [I] --------- Verifying Ranger DB tables ---------
2026-08-10 22:35:27,571 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-10 22:35:27,571 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfcURSo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.84:57811/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-10 22:35:27,818 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-10 22:35:27,819 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 22:35:27,819 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-10 22:35:27,819 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfcURSo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.84:57811/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-10 22:35:28,106 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260810 22:35:28.661958 17317 mini_ranger_kms.cc:310] Created kms service
I20260810 22:35:28.891178 17317 mini_ranger_kms.cc:326] Created kudu user
I20260810 22:35:28.942884 17317 mini_ranger_kms.cc:343] Created rangerkms user
Aug 10 22:35:28 dist-test-slave-77v9 krb5kdc[29371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401325, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.16.233.84@KRBTEST.COM
I20260810 22:35:29.377391 17317 mini_ranger_kms.cc:384] Added ranger policy
I20260810 22:35:29.377568 17317 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/ranger-kms:/tmp/dist-test-taskfcURSo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfcURSo/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskfcURSo/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260810 22:35:29.377595 17317 mini_ranger_kms.cc:240] Using host: 127.16.233.84
I20260810 22:35:29.383270 17317 mini_ranger_kms.cc:276] Ranger KMS PID: 29846
I20260810 22:35:29.383463 17317 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.16.233.84:50379
Aug 10, 2026 10:35:30 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 10:35:30 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 10:35:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 10, 2026 10:35:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 10, 2026 10:35:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/ranger-kms/ews/webapp].
Aug 10, 2026 10:35:31 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260810 22:35:31.970463 17317 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.586s user 0.000s sys 0.705s
I20260810 22:35:31.970646 17317 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260810 22:35:31.970781 17317 mini_ranger_kms.cc:401] 127.16.233.84:50379/kms/v1/keys
Aug 10 22:35:32 dist-test-slave-77v9 krb5kdc[29371](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786401332, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.233.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 10 22:35:32 dist-test-slave-77v9 krb5kdc[29371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401332, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.233.84@KRBTEST.COM for HTTP/127.16.233.84@KRBTEST.COM
Aug 10 22:35:33 dist-test-slave-77v9 krb5kdc[29371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401332, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.233.84@KRBTEST.COM for HTTP/127.16.233.84@KRBTEST.COM
Aug 10 22:35:35 dist-test-slave-77v9 krb5kdc[29371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401335, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.16.233.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab.
Entry for principal kudu/127.16.233.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab.
I20260810 22:35:35.876333 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:44051
--webserver_interface=127.16.233.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab
--principal=kudu/127.16.233.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:45779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.126:44051
--ranger_config_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.233.84:50379/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:35:36.049072 29903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:35:36.049374 29903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:35:36.049423 29903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:35:36.053660 29903 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:35:36.053732 29903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:35:36.053751 29903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:35:36.053768 29903 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:35:36.053783 29903 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:35:36.059657 29903 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:45779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/master-0/wal
--ranger_kms_url=127.16.233.84:50379/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.16.233.126:44051
--ranger_config_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab
--principal=kudu/127.16.233.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:44051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:35:36.060777 29903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:35:36.061949 29903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:35:36.072888 29909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:35:36.072927 29910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:35:36.072952 29903 server_base.cc:1034] running on GCE node
W20260810 22:35:36.072784 29912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:35:36.073714 29903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:35:36.075043 29903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:35:36.076248 29903 hybrid_clock.cc:648] HybridClock initialized: now 1786401336076211 us; error 53 us; skew 500 ppm
Aug 10 22:35:36 dist-test-slave-77v9 krb5kdc[29371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401336, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:35:36.085202 29903 init.cc:377] Logged in from keytab as kudu/127.16.233.126@KRBTEST.COM (short username kudu)
I20260810 22:35:36.086373 29903 webserver.cc:460] Webserver started at http://127.16.233.126:43913/ using document root <none> and password file <none>
I20260810 22:35:36.087064 29903 fs_manager.cc:359] Metadata directory not provided
I20260810 22:35:36.087153 29903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:35:36.087393 29903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 22:35:36 dist-test-slave-77v9 krb5kdc[29371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401336, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.126@KRBTEST.COM for HTTP/127.16.233.84@KRBTEST.COM
I20260810 22:35:36.161501 29903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/master-0/data/instance:
uuid: "6fb71a43b486495f810001168e757d58"
format_stamp: "Formatted at 2026-08-10 22:35:36 on dist-test-slave-77v9"
server_key: "424b742bb07da64835a2e9fb9637ed86"
server_key_iv: "ab984086f2590f062fc6450fb9de0d02"
server_key_version: "kuduclusterkey@0"
I20260810 22:35:36.162231 29903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/master-0/wal/instance:
uuid: "6fb71a43b486495f810001168e757d58"
format_stamp: "Formatted at 2026-08-10 22:35:36 on dist-test-slave-77v9"
server_key: "424b742bb07da64835a2e9fb9637ed86"
server_key_iv: "ab984086f2590f062fc6450fb9de0d02"
server_key_version: "kuduclusterkey@0"
I20260810 22:35:36.166659 29903 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 22:35:36.187170 29921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:35:36.189100 29903 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260810 22:35:36.189275 29903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/master-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/master-0/wal
uuid: "6fb71a43b486495f810001168e757d58"
format_stamp: "Formatted at 2026-08-10 22:35:36 on dist-test-slave-77v9"
server_key: "424b742bb07da64835a2e9fb9637ed86"
server_key_iv: "ab984086f2590f062fc6450fb9de0d02"
server_key_version: "kuduclusterkey@0"
I20260810 22:35:36.189430 29903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:35:36.228370 29903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:35:36.232065 29903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:35:36.232362 29903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:35:36.245267 29903 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.126:44051
I20260810 22:35:36.245317 29977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.126:44051 every 8 connection(s)
I20260810 22:35:36.246354 29903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
I20260810 22:35:36.248950 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 29903
I20260810 22:35:36.249083 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/master-0/wal/instance
I20260810 22:35:36.249390 17317 external_mini_cluster.cc:1428] Setting key 68615e019a578c621f88c3d1bc1dc7ac
I20260810 22:35:36.249655 29979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:35:36.258183 29979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fb71a43b486495f810001168e757d58: Bootstrap starting.
I20260810 22:35:36.261308 29979 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6fb71a43b486495f810001168e757d58: Neither blocks nor log segments found. Creating new log.
Aug 10 22:35:36 dist-test-slave-77v9 krb5kdc[29371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401335, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:35:36.262740 29979 log.cc:824] T 00000000000000000000000000000000 P 6fb71a43b486495f810001168e757d58: Log is configured to *not* fsync() on all Append() calls
I20260810 22:35:36.266224 29979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fb71a43b486495f810001168e757d58: No bootstrap required, opened a new log
I20260810 22:35:36.269083 29982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:35:36.250962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42292 (local address 127.16.233.126:44051)
0810 22:35:36.251608 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0810 22:35:36.251636 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0810 22:35:36.251703 (+ 67us) server_negotiation.cc:407] Connection header received
0810 22:35:36.252917 (+ 1214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:35:36.252945 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:35:36.253423 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:35:36.253807 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:35:36.254817 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:35:36.256660 (+ 1843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:35:36.258010 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:35:36.258765 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:35:36.258925 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:35:36.262706 (+ 3781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:35:36.262713 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:35:36.262732 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:35:36.262769 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:35:36.265519 (+ 2750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:35:36.266028 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:35:36.266033 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:35:36.266039 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:35:36.266181 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:35:36.266514 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:35:36.266518 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:35:36.266520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:35:36.266926 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:35:36.267192 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0810 22:35:36.267878 (+ 686us) server_negotiation.cc:299] Negotiation successful
0810 22:35:36.268199 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":289,"threads_started":1}
I20260810 22:35:36.271412 29979 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6fb71a43b486495f810001168e757d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fb71a43b486495f810001168e757d58" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 44051 } }
I20260810 22:35:36.271708 29979 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6fb71a43b486495f810001168e757d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:35:36.271768 29979 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6fb71a43b486495f810001168e757d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fb71a43b486495f810001168e757d58, State: Initialized, Role: FOLLOWER
I20260810 22:35:36.272241 29979 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fb71a43b486495f810001168e757d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fb71a43b486495f810001168e757d58" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 44051 } }
I20260810 22:35:36.272413 29979 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6fb71a43b486495f810001168e757d58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:35:36.272475 29979 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6fb71a43b486495f810001168e757d58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:35:36.272569 29979 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fb71a43b486495f810001168e757d58 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:35:36.274047 29979 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6fb71a43b486495f810001168e757d58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fb71a43b486495f810001168e757d58" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 44051 } }
I20260810 22:35:36.274456 29979 leader_election.cc:302] T 00000000000000000000000000000000 P 6fb71a43b486495f810001168e757d58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6fb71a43b486495f810001168e757d58; no voters:
I20260810 22:35:36.274830 29979 leader_election.cc:288] T 00000000000000000000000000000000 P 6fb71a43b486495f810001168e757d58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:35:36.275192 29984 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6fb71a43b486495f810001168e757d58 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:35:36.275642 29984 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6fb71a43b486495f810001168e757d58 [term 1 LEADER]: Becoming Leader. State: Replica: 6fb71a43b486495f810001168e757d58, State: Running, Role: LEADER
I20260810 22:35:36.276033 29984 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6fb71a43b486495f810001168e757d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fb71a43b486495f810001168e757d58" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 44051 } }
I20260810 22:35:36.276283 29979 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6fb71a43b486495f810001168e757d58 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:35:36.278743 29985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fb71a43b486495f810001168e757d58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6fb71a43b486495f810001168e757d58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fb71a43b486495f810001168e757d58" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 44051 } } }
I20260810 22:35:36.278894 29985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fb71a43b486495f810001168e757d58 [sys.catalog]: This master's current role is: LEADER
I20260810 22:35:36.279387 30002 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:35:36.279330 29986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fb71a43b486495f810001168e757d58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fb71a43b486495f810001168e757d58. Latest consensus state: current_term: 1 leader_uuid: "6fb71a43b486495f810001168e757d58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fb71a43b486495f810001168e757d58" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 44051 } } }
I20260810 22:35:36.279496 29986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fb71a43b486495f810001168e757d58 [sys.catalog]: This master's current role is: LEADER
I20260810 22:35:36.279542 29979 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 22:35:36.288846 30002 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:35:36.299302 30002 catalog_manager.cc:1347] Generated new cluster ID: a48a23c9378548b5a3f95ad52c6a5c24
I20260810 22:35:36.299381 30002 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:35:36.317540 30002 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:35:36.318964 30002 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:35:36.329528 30002 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6fb71a43b486495f810001168e757d58: Generated new TSK 0
I20260810 22:35:36.330631 30002 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.233.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab.
Entry for principal kudu/127.16.233.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab.
I20260810 22:35:37.361629 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.233.84:50379/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.65:0
--local_ip_for_outbound_sockets=127.16.233.65
--webserver_interface=127.16.233.65
--webserver_port=0
--tserver_master_addrs=127.16.233.126:44051
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab
--principal=kudu/127.16.233.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:45779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:35:37.556078 30030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:35:37.556535 30030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:35:37.556661 30030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:35:37.561149 30030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:35:37.561219 30030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:35:37.561318 30030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.65
I20260810 22:35:37.567215 30030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:45779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.16.233.84:50379/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab
--principal=kudu/127.16.233.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.233.65
--webserver_port=0
--tserver_master_addrs=127.16.233.126:44051
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.65
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:35:37.568472 30030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:35:37.569643 30030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:35:37.583112 30036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:35:37.583150 30038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:35:37.583452 30035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:35:37.584429 30030 server_base.cc:1034] running on GCE node
I20260810 22:35:37.585070 30030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:35:37.586000 30030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:35:37.589327 30030 hybrid_clock.cc:648] HybridClock initialized: now 1786401337589267 us; error 63 us; skew 500 ppm
Aug 10 22:35:37 dist-test-slave-77v9 krb5kdc[29371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401337, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:35:37.594213 30030 init.cc:377] Logged in from keytab as kudu/127.16.233.65@KRBTEST.COM (short username kudu)
I20260810 22:35:37.595662 30030 webserver.cc:460] Webserver started at http://127.16.233.65:39549/ using document root <none> and password file <none>
I20260810 22:35:37.596632 30030 fs_manager.cc:359] Metadata directory not provided
I20260810 22:35:37.596807 30030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:35:37.597242 30030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 22:35:37 dist-test-slave-77v9 krb5kdc[29371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401337, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.65@KRBTEST.COM for HTTP/127.16.233.84@KRBTEST.COM
I20260810 22:35:37.617486 30030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/ts-0/data/instance:
uuid: "39404e54bda54b45a85caac3d9f5b935"
format_stamp: "Formatted at 2026-08-10 22:35:37 on dist-test-slave-77v9"
server_key: "1f97b27e5fb6de8a83e2c4e315b032ba"
server_key_iv: "a5e0972fbba8d30e91a8848077c95645"
server_key_version: "kuduclusterkey@0"
I20260810 22:35:37.618589 30030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/ts-0/wal/instance:
uuid: "39404e54bda54b45a85caac3d9f5b935"
format_stamp: "Formatted at 2026-08-10 22:35:37 on dist-test-slave-77v9"
server_key: "1f97b27e5fb6de8a83e2c4e315b032ba"
server_key_iv: "a5e0972fbba8d30e91a8848077c95645"
server_key_version: "kuduclusterkey@0"
I20260810 22:35:37.625267 30030 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260810 22:35:37.645329 30048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:35:37.647572 30030 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260810 22:35:37.647751 30030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/ts-0/wal
uuid: "39404e54bda54b45a85caac3d9f5b935"
format_stamp: "Formatted at 2026-08-10 22:35:37 on dist-test-slave-77v9"
server_key: "1f97b27e5fb6de8a83e2c4e315b032ba"
server_key_iv: "a5e0972fbba8d30e91a8848077c95645"
server_key_version: "kuduclusterkey@0"
I20260810 22:35:37.647939 30030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:35:37.695739 30030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:35:37.700215 30030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:35:37.700582 30030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:35:37.701604 30030 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:35:37.702880 30030 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:35:37.702999 30030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:35:37.703155 30030 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:35:37.703270 30030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:35:37.726107 30030 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.65:44039
I20260810 22:35:37.727548 30030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
I20260810 22:35:37.734999 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 30030
I20260810 22:35:37.735141 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/ts-0/wal/instance
I20260810 22:35:37.735460 17317 external_mini_cluster.cc:1428] Setting key 35bd9854759cf4a0a9c8eec93f9a1890
I20260810 22:35:37.739135 30178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.65:44039 every 8 connection(s)
WARNING: no policy specified for kudu/127.16.233.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.66@KRBTEST.COM" created.
Aug 10 22:35:37 dist-test-slave-77v9 krb5kdc[29371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401337, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.65@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:35:37.778347 30183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:35:37.743349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:46459 (local address 127.16.233.126:44051)
0810 22:35:37.756761 (+ 13412us) server_negotiation.cc:206] Beginning negotiation
0810 22:35:37.756766 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:35:37.756787 (+ 21us) server_negotiation.cc:407] Connection header received
0810 22:35:37.756867 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:35:37.756870 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:35:37.756936 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:35:37.757188 (+ 252us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:35:37.761662 (+ 4474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:35:37.762595 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:35:37.766738 (+ 4143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:35:37.767184 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:35:37.767317 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:35:37.771399 (+ 4082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:35:37.771405 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:35:37.771408 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:35:37.771444 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:35:37.773923 (+ 2479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:35:37.774853 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:35:37.774857 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:35:37.774860 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:35:37.774976 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:35:37.775529 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:35:37.775533 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:35:37.775535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:35:37.775879 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:35:37.776001 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0810 22:35:37.777154 (+ 1153us) server_negotiation.cc:299] Negotiation successful
0810 22:35:37.778013 (+ 859us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13295,"thread_start_us":74,"threads_started":1}
I20260810 22:35:37.780509 30182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:35:37.741920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:44051 (local address 127.16.233.65:46459)
0810 22:35:37.747915 (+ 5995us) negotiation.cc:107] Waiting for socket to connect
0810 22:35:37.748001 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0810 22:35:37.754672 (+ 6671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:35:37.757586 (+ 2914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:35:37.757611 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:35:37.760372 (+ 2761us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:35:37.761429 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:35:37.761455 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:35:37.764520 (+ 3065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:35:37.764528 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:35:37.766505 (+ 1977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:35:37.766523 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:35:37.767556 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:35:37.767563 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:35:37.767795 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:35:37.768885 (+ 1090us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:35:37.768914 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:35:37.771126 (+ 2212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:35:37.774136 (+ 3010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:35:37.774145 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:35:37.774165 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:35:37.774652 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:35:37.775125 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:35:37.775129 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:35:37.775131 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:35:37.775366 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:35:37.776074 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:35:37.776083 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:35:37.776487 (+ 404us) client_negotiation.cc:769] Sending connection context
0810 22:35:37.776755 (+ 268us) client_negotiation.cc:240] Negotiation successful
0810 22:35:37.779220 (+ 2465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5668,"thread_start_us":293,"threads_started":1}
I20260810 22:35:37.782828 30179 heartbeater.cc:347] Connected to a master server at 127.16.233.126:44051
I20260810 22:35:37.783149 30179 heartbeater.cc:464] Registering TS with master...
I20260810 22:35:37.784128 30179 heartbeater.cc:511] Master 127.16.233.126:44051 requested a full tablet report, sending...
I20260810 22:35:37.786525 29938 ts_manager.cc:194] Registered new tserver with Master: 39404e54bda54b45a85caac3d9f5b935 (127.16.233.65:44039)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab.
Entry for principal kudu/127.16.233.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab.
I20260810 22:35:37.788386 29938 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.65@KRBTEST.COM'} at 127.16.233.65:46459
I20260810 22:35:37.789053 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.233.84:50379/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.66:0
--local_ip_for_outbound_sockets=127.16.233.66
--webserver_interface=127.16.233.66
--webserver_port=0
--tserver_master_addrs=127.16.233.126:44051
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab
--principal=kudu/127.16.233.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:45779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 22:35:37 dist-test-slave-77v9 krb5kdc[29371](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786401337, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260810 22:35:38.020433 30189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:35:38.020992 30189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:35:38.021124 30189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:35:38.027518 30189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:35:38.027688 30189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:35:38.027843 30189 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.66
I20260810 22:35:38.036299 30189 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:45779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.16.233.84:50379/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab
--principal=kudu/127.16.233.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.233.66
--webserver_port=0
--tserver_master_addrs=127.16.233.126:44051
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30189
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.66
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:35:38.037793 30189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:35:38.039216 30189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:35:38.052459 30201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:35:38.052747 30199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:35:38.052533 30198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:35:38.054503 30189 server_base.cc:1034] running on GCE node
I20260810 22:35:38.055065 30189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:35:38.055959 30189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:35:38.059293 30189 hybrid_clock.cc:648] HybridClock initialized: now 1786401338059135 us; error 169 us; skew 500 ppm
Aug 10 22:35:38 dist-test-slave-77v9 krb5kdc[29371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401338, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:35:38.064435 30189 init.cc:377] Logged in from keytab as kudu/127.16.233.66@KRBTEST.COM (short username kudu)
I20260810 22:35:38.065961 30189 webserver.cc:460] Webserver started at http://127.16.233.66:33249/ using document root <none> and password file <none>
I20260810 22:35:38.066816 30189 fs_manager.cc:359] Metadata directory not provided
I20260810 22:35:38.066934 30189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:35:38.067282 30189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 22:35:38 dist-test-slave-77v9 krb5kdc[29371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401338, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.66@KRBTEST.COM for HTTP/127.16.233.84@KRBTEST.COM
I20260810 22:35:38.089037 30189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/ts-1/data/instance:
uuid: "85ffe7674ce641d5ad5df9cda594e5cf"
format_stamp: "Formatted at 2026-08-10 22:35:38 on dist-test-slave-77v9"
server_key: "43cf33fae45e14574f1589990d2394b6"
server_key_iv: "a27b9c132eb9e9571c07c1072e72eb65"
server_key_version: "kuduclusterkey@0"
I20260810 22:35:38.090243 30189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/ts-1/wal/instance:
uuid: "85ffe7674ce641d5ad5df9cda594e5cf"
format_stamp: "Formatted at 2026-08-10 22:35:38 on dist-test-slave-77v9"
server_key: "43cf33fae45e14574f1589990d2394b6"
server_key_iv: "a27b9c132eb9e9571c07c1072e72eb65"
server_key_version: "kuduclusterkey@0"
I20260810 22:35:38.095290 30189 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 22:35:38.114610 30208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:35:38.116312 30189 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260810 22:35:38.116485 30189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/ts-1/wal
uuid: "85ffe7674ce641d5ad5df9cda594e5cf"
format_stamp: "Formatted at 2026-08-10 22:35:38 on dist-test-slave-77v9"
server_key: "43cf33fae45e14574f1589990d2394b6"
server_key_iv: "a27b9c132eb9e9571c07c1072e72eb65"
server_key_version: "kuduclusterkey@0"
I20260810 22:35:38.116660 30189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:35:38.144153 30189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:35:38.148029 30189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:35:38.148308 30189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:35:38.149261 30189 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:35:38.150445 30189 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:35:38.150527 30189 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:35:38.150635 30189 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:35:38.150686 30189 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:35:38.185745 30189 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.66:46035
I20260810 22:35:38.187314 30189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
I20260810 22:35:38.188314 30337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.66:46035 every 8 connection(s)
I20260810 22:35:38.197470 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 30189
I20260810 22:35:38.197618 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/ts-1/wal/instance
I20260810 22:35:38.198046 17317 external_mini_cluster.cc:1428] Setting key 69e519d0ce743e7d653fa3b32709be9c
Aug 10 22:35:38 dist-test-slave-77v9 krb5kdc[29371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401338, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.66@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:35:38.213595 30183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:35:38.191888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:41273 (local address 127.16.233.126:44051)
0810 22:35:38.192052 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 22:35:38.192057 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:35:38.192094 (+ 37us) server_negotiation.cc:407] Connection header received
0810 22:35:38.193107 (+ 1013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:35:38.193111 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:35:38.193171 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:35:38.193280 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:35:38.194847 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:35:38.195522 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:35:38.197497 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:35:38.197852 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:35:38.197941 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:35:38.207537 (+ 9596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:35:38.207545 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:35:38.207551 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:35:38.207600 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:35:38.209825 (+ 2225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:35:38.210894 (+ 1069us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:35:38.210898 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:35:38.210900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:35:38.211001 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:35:38.211552 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:35:38.211556 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:35:38.211558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:35:38.211851 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:35:38.211948 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 22:35:38.213083 (+ 1135us) server_negotiation.cc:299] Negotiation successful
0810 22:35:38.213290 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
WARNING: no policy specified for kudu/127.16.233.67@KRBTEST.COM; defaulting to no policy
I20260810 22:35:38.215351 30341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:35:38.190728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:44051 (local address 127.16.233.66:41273)
0810 22:35:38.191428 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0810 22:35:38.191484 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 22:35:38.192863 (+ 1379us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:35:38.193461 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:35:38.193468 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:35:38.193987 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:35:38.194684 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:35:38.194701 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:35:38.195661 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:35:38.195664 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:35:38.197321 (+ 1657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:35:38.197336 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:35:38.198169 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:35:38.198173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:35:38.198509 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:35:38.199409 (+ 900us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:35:38.199430 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:35:38.201619 (+ 2189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:35:38.210064 (+ 8445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:35:38.210071 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:35:38.210094 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:35:38.210745 (+ 651us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:35:38.211157 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:35:38.211160 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:35:38.211162 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:35:38.211419 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:35:38.212029 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:35:38.212038 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:35:38.212676 (+ 638us) client_negotiation.cc:769] Sending connection context
0810 22:35:38.213809 (+ 1133us) client_negotiation.cc:240] Negotiation successful
0810 22:35:38.214196 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":218,"threads_started":1}
I20260810 22:35:38.217113 30338 heartbeater.cc:347] Connected to a master server at 127.16.233.126:44051
I20260810 22:35:38.217394 30338 heartbeater.cc:464] Registering TS with master...
I20260810 22:35:38.217969 30338 heartbeater.cc:511] Master 127.16.233.126:44051 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.67@KRBTEST.COM" created.
I20260810 22:35:38.219172 29938 ts_manager.cc:194] Registered new tserver with Master: 85ffe7674ce641d5ad5df9cda594e5cf (127.16.233.66:46035)
I20260810 22:35:38.219784 29938 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.66@KRBTEST.COM'} at 127.16.233.66:41273
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab.
Entry for principal kudu/127.16.233.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab.
I20260810 22:35:38.245298 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.233.84:50379/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.67:0
--local_ip_for_outbound_sockets=127.16.233.67
--webserver_interface=127.16.233.67
--webserver_port=0
--tserver_master_addrs=127.16.233.126:44051
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab
--principal=kudu/127.16.233.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:45779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:35:38.478574 30345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:35:38.479269 30345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:35:38.479417 30345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:35:38.486415 30345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:35:38.486646 30345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:35:38.486837 30345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.67
I20260810 22:35:38.496490 30345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:45779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.16.233.84:50379/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab
--principal=kudu/127.16.233.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.233.67
--webserver_port=0
--tserver_master_addrs=127.16.233.126:44051
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30345
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.67
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:35:38.498575 30345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:35:38.500130 30345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:35:38.515771 30352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:35:38.516360 30351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:35:38.517257 30354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:35:38.518373 30345 server_base.cc:1034] running on GCE node
I20260810 22:35:38.519065 30345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:35:38.519886 30345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:35:38.521039 30345 hybrid_clock.cc:648] HybridClock initialized: now 1786401338520966 us; error 88 us; skew 500 ppm
Aug 10 22:35:38 dist-test-slave-77v9 krb5kdc[29371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401338, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:35:38.526193 30345 init.cc:377] Logged in from keytab as kudu/127.16.233.67@KRBTEST.COM (short username kudu)
I20260810 22:35:38.527400 30345 webserver.cc:460] Webserver started at http://127.16.233.67:40301/ using document root <none> and password file <none>
I20260810 22:35:38.528160 30345 fs_manager.cc:359] Metadata directory not provided
I20260810 22:35:38.528308 30345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:35:38.528606 30345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 22:35:38 dist-test-slave-77v9 krb5kdc[29371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401338, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.67@KRBTEST.COM for HTTP/127.16.233.84@KRBTEST.COM
I20260810 22:35:38.548578 30345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/ts-2/data/instance:
uuid: "e7497bb27dae4d708948b3e78421e576"
format_stamp: "Formatted at 2026-08-10 22:35:38 on dist-test-slave-77v9"
server_key: "6b3defd75032d2ec3b602a22969a9c7d"
server_key_iv: "55e083085c23c60970f04d40eb164d0a"
server_key_version: "kuduclusterkey@0"
I20260810 22:35:38.550575 30345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/ts-2/wal/instance:
uuid: "e7497bb27dae4d708948b3e78421e576"
format_stamp: "Formatted at 2026-08-10 22:35:38 on dist-test-slave-77v9"
server_key: "6b3defd75032d2ec3b602a22969a9c7d"
server_key_iv: "55e083085c23c60970f04d40eb164d0a"
server_key_version: "kuduclusterkey@0"
I20260810 22:35:38.567031 30345 fs_manager.cc:691] Time spent creating directory manager: real 0.016s user 0.002s sys 0.003s
I20260810 22:35:38.589293 30361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:35:38.591123 30345 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260810 22:35:38.591269 30345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/ts-2/wal
uuid: "e7497bb27dae4d708948b3e78421e576"
format_stamp: "Formatted at 2026-08-10 22:35:38 on dist-test-slave-77v9"
server_key: "6b3defd75032d2ec3b602a22969a9c7d"
server_key_iv: "55e083085c23c60970f04d40eb164d0a"
server_key_version: "kuduclusterkey@0"
I20260810 22:35:38.591435 30345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:35:38.607411 30345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:35:38.611912 30345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:35:38.612221 30345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:35:38.613209 30345 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:35:38.614771 30345 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:35:38.614893 30345 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:35:38.615010 30345 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:35:38.615079 30345 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:35:38.646659 30345 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.67:35091
I20260810 22:35:38.648314 30345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
I20260810 22:35:38.651415 30492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.67:35091 every 8 connection(s)
I20260810 22:35:38.655697 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 30345
I20260810 22:35:38.655853 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/ts-2/wal/instance
I20260810 22:35:38.656230 17317 external_mini_cluster.cc:1428] Setting key 4117c5fd7a18f8c6114a0008bcb0b657
Aug 10 22:35:38 dist-test-slave-77v9 krb5kdc[29371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401338, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.67@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:35:38.687826 30183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:35:38.656389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:39657 (local address 127.16.233.126:44051)
0810 22:35:38.656537 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 22:35:38.656543 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:35:38.665607 (+ 9064us) server_negotiation.cc:407] Connection header received
0810 22:35:38.667810 (+ 2203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:35:38.667814 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:35:38.667885 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:35:38.667950 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:35:38.670561 (+ 2611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:35:38.671504 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:35:38.676502 (+ 4998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:35:38.676926 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:35:38.677020 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:35:38.681884 (+ 4864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:35:38.681888 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:35:38.681891 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:35:38.681924 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:35:38.683844 (+ 1920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:35:38.684606 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:35:38.684609 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:35:38.684611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:35:38.684729 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:35:38.686315 (+ 1586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:35:38.686319 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:35:38.686321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:35:38.686631 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:35:38.686726 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 22:35:38.687519 (+ 793us) server_negotiation.cc:299] Negotiation successful
0810 22:35:38.687662 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 22:35:38.690218 30496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:35:38.654965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:44051 (local address 127.16.233.67:39657)
0810 22:35:38.665373 (+ 10408us) negotiation.cc:107] Waiting for socket to connect
0810 22:35:38.665472 (+ 99us) client_negotiation.cc:174] Beginning negotiation
0810 22:35:38.667479 (+ 2007us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:35:38.668476 (+ 997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:35:38.668487 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:35:38.669274 (+ 787us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:35:38.670333 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:35:38.670360 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:35:38.671941 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:35:38.671945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:35:38.676322 (+ 4377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:35:38.676340 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:35:38.677321 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:35:38.677326 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:35:38.677719 (+ 393us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:35:38.678817 (+ 1098us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:35:38.678843 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:35:38.681632 (+ 2789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:35:38.684014 (+ 2382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:35:38.684022 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:35:38.684042 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:35:38.684475 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:35:38.684876 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:35:38.684882 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:35:38.684884 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:35:38.686181 (+ 1297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:35:38.686801 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:35:38.686809 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:35:38.687183 (+ 374us) client_negotiation.cc:769] Sending connection context
0810 22:35:38.688334 (+ 1151us) client_negotiation.cc:240] Negotiation successful
0810 22:35:38.688789 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10064,"thread_start_us":303,"threads_started":1}
I20260810 22:35:38.692482 30494 heartbeater.cc:347] Connected to a master server at 127.16.233.126:44051
I20260810 22:35:38.692780 30494 heartbeater.cc:464] Registering TS with master...
I20260810 22:35:38.693431 30494 heartbeater.cc:511] Master 127.16.233.126:44051 requested a full tablet report, sending...
I20260810 22:35:38.695575 29938 ts_manager.cc:194] Registered new tserver with Master: e7497bb27dae4d708948b3e78421e576 (127.16.233.67:35091)
I20260810 22:35:38.696439 29938 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.67@KRBTEST.COM'} at 127.16.233.67:39657
I20260810 22:35:38.699692 17317 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 22:35:38.714510 30183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:35:38.702547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42296 (local address 127.16.233.126:44051)
0810 22:35:38.702676 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 22:35:38.702681 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:35:38.703359 (+ 678us) server_negotiation.cc:407] Connection header received
0810 22:35:38.703569 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:35:38.703573 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:35:38.703624 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:35:38.703679 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:35:38.704860 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:35:38.705786 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:35:38.707117 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:35:38.707472 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:35:38.707561 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:35:38.709937 (+ 2376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:35:38.709941 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:35:38.709944 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:35:38.709980 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:35:38.712041 (+ 2061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:35:38.712718 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:35:38.712722 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:35:38.712724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:35:38.712870 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:35:38.713302 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:35:38.713306 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:35:38.713308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:35:38.713583 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:35:38.713701 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 22:35:38.714144 (+ 443us) server_negotiation.cc:299] Negotiation successful
0810 22:35:38.714316 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 22:35:38.720145 29938 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42296:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 22:35:38.725651 29938 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Aug 10 22:35:38 dist-test-slave-77v9 krb5kdc[29371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401337, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.126@KRBTEST.COM for HTTP/127.16.233.84@KRBTEST.COM
I20260810 22:35:38.757565 30508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:35:38.740636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.67:35091 (local address 127.0.0.1:50138)
0810 22:35:38.746838 (+ 6202us) negotiation.cc:107] Waiting for socket to connect
0810 22:35:38.746907 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0810 22:35:38.747060 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:35:38.749612 (+ 2552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:35:38.749618 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:35:38.749639 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:35:38.749804 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:35:38.749810 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:35:38.751572 (+ 1762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:35:38.751576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:35:38.754353 (+ 2777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:35:38.754368 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:35:38.756746 (+ 2378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:35:38.756785 (+ 39us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:35:38.757066 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:35:38.757118 (+ 52us) client_negotiation.cc:769] Sending connection context
0810 22:35:38.757209 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 22:35:38.757387 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5979,"mutex_wait_us":104,"thread_start_us":125,"threads_started":1}
I20260810 22:35:38.758632 30509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:35:38.742144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50138 (local address 127.16.233.67:35091)
0810 22:35:38.747741 (+ 5597us) server_negotiation.cc:206] Beginning negotiation
0810 22:35:38.747754 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 22:35:38.747783 (+ 29us) server_negotiation.cc:407] Connection header received
0810 22:35:38.747867 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:35:38.747872 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:35:38.748116 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:35:38.749952 (+ 1836us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:35:38.750126 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:35:38.751416 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:35:38.754653 (+ 3237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:35:38.755553 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:35:38.755672 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:35:38.758309 (+ 2637us) server_negotiation.cc:1035] Waiting for connection context
0810 22:35:38.758372 (+ 63us) server_negotiation.cc:299] Negotiation successful
0810 22:35:38.758481 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5471,"thread_start_us":143,"threads_started":1}
I20260810 22:35:38.762197 30410 tablet_service.cc:1467] Processing CreateTablet for tablet f63d88532134472d9b468345f07f24b4 (DEFAULT_TABLE table=test-table [id=e7e1a0177ba045e187b6e7c07de4f1fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:35:38.763515 30410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f63d88532134472d9b468345f07f24b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:35:38.770685 30510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:35:38.744925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.65:44039 (local address 127.0.0.1:43698)
0810 22:35:38.749372 (+ 4447us) negotiation.cc:107] Waiting for socket to connect
0810 22:35:38.749388 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 22:35:38.749479 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:35:38.756734 (+ 7255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:35:38.756738 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:35:38.756759 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:35:38.756939 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:35:38.756947 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:35:38.765460 (+ 8513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:35:38.765464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:35:38.766816 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:35:38.766830 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:35:38.770218 (+ 3388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:35:38.770225 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:35:38.770353 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:35:38.770387 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 22:35:38.770446 (+ 59us) client_negotiation.cc:240] Negotiation successful
0810 22:35:38.770527 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4371,"mutex_wait_us":1,"thread_start_us":110,"threads_started":1}
I20260810 22:35:38.777045 30514 tablet_bootstrap.cc:492] T f63d88532134472d9b468345f07f24b4 P e7497bb27dae4d708948b3e78421e576: Bootstrap starting.
I20260810 22:35:38.777630 30513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:35:38.745106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43698 (local address 127.16.233.65:44039)
0810 22:35:38.755739 (+ 10633us) server_negotiation.cc:206] Beginning negotiation
0810 22:35:38.755788 (+ 49us) server_negotiation.cc:399] Waiting for connection header
0810 22:35:38.755818 (+ 30us) server_negotiation.cc:407] Connection header received
0810 22:35:38.755906 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:35:38.755914 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:35:38.756179 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:35:38.762738 (+ 6559us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:35:38.762991 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:35:38.764579 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:35:38.766998 (+ 2419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:35:38.767890 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:35:38.776975 (+ 9085us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:35:38.777204 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0810 22:35:38.777290 (+ 86us) server_negotiation.cc:299] Negotiation successful
0810 22:35:38.777411 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10498,"thread_start_us":159,"threads_started":1}
I20260810 22:35:38.782536 30514 tablet_bootstrap.cc:654] T f63d88532134472d9b468345f07f24b4 P e7497bb27dae4d708948b3e78421e576: Neither blocks nor log segments found. Creating new log.
I20260810 22:35:38.781919 30101 tablet_service.cc:1467] Processing CreateTablet for tablet f63d88532134472d9b468345f07f24b4 (DEFAULT_TABLE table=test-table [id=e7e1a0177ba045e187b6e7c07de4f1fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:35:38.783563 30101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f63d88532134472d9b468345f07f24b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:35:38.784216 30514 log.cc:824] T f63d88532134472d9b468345f07f24b4 P e7497bb27dae4d708948b3e78421e576: Log is configured to *not* fsync() on all Append() calls
I20260810 22:35:38.787191 30514 tablet_bootstrap.cc:492] T f63d88532134472d9b468345f07f24b4 P e7497bb27dae4d708948b3e78421e576: No bootstrap required, opened a new log
I20260810 22:35:38.787515 30514 ts_tablet_manager.cc:1397] T f63d88532134472d9b468345f07f24b4 P e7497bb27dae4d708948b3e78421e576: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.002s
I20260810 22:35:38.792348 30179 heartbeater.cc:503] Master 127.16.233.126:44051 was elected leader, sending a full tablet report...
I20260810 22:35:38.795300 30514 raft_consensus.cc:348] T f63d88532134472d9b468345f07f24b4 P e7497bb27dae4d708948b3e78421e576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7497bb27dae4d708948b3e78421e576" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 35091 } } peers { permanent_uuid: "85ffe7674ce641d5ad5df9cda594e5cf" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 46035 } } peers { permanent_uuid: "39404e54bda54b45a85caac3d9f5b935" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 44039 } }
I20260810 22:35:38.795883 30514 raft_consensus.cc:374] T f63d88532134472d9b468345f07f24b4 P e7497bb27dae4d708948b3e78421e576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:35:38.795943 30512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:35:38.743952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.66:46035 (local address 127.0.0.1:60750)
0810 22:35:38.747358 (+ 3406us) negotiation.cc:107] Waiting for socket to connect
0810 22:35:38.747372 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:35:38.747454 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:35:38.768090 (+ 20636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:35:38.768094 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:35:38.768121 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:35:38.768407 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:35:38.768419 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:35:38.789063 (+ 20644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:35:38.789069 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:35:38.790748 (+ 1679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:35:38.790769 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:35:38.795326 (+ 4557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:35:38.795333 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:35:38.795546 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:35:38.795579 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 22:35:38.795651 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 22:35:38.795773 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3334,"thread_start_us":108,"threads_started":1}
I20260810 22:35:38.796021 30514 raft_consensus.cc:729] T f63d88532134472d9b468345f07f24b4 P e7497bb27dae4d708948b3e78421e576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7497bb27dae4d708948b3e78421e576, State: Initialized, Role: FOLLOWER
I20260810 22:35:38.796700 30514 consensus_queue.cc:260] T f63d88532134472d9b468345f07f24b4 P e7497bb27dae4d708948b3e78421e576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7497bb27dae4d708948b3e78421e576" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 35091 } } peers { permanent_uuid: "85ffe7674ce641d5ad5df9cda594e5cf" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 46035 } } peers { permanent_uuid: "39404e54bda54b45a85caac3d9f5b935" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 44039 } }
I20260810 22:35:38.798185 30514 ts_tablet_manager.cc:1428] T f63d88532134472d9b468345f07f24b4 P e7497bb27dae4d708948b3e78421e576: Time spent starting tablet: real 0.011s user 0.008s sys 0.001s
I20260810 22:35:38.798642 30494 heartbeater.cc:503] Master 127.16.233.126:44051 was elected leader, sending a full tablet report...
I20260810 22:35:38.799980 30511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:35:38.744143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60750 (local address 127.16.233.66:46035)
0810 22:35:38.764940 (+ 20797us) server_negotiation.cc:206] Beginning negotiation
0810 22:35:38.764947 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 22:35:38.764973 (+ 26us) server_negotiation.cc:407] Connection header received
0810 22:35:38.765053 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:35:38.765058 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:35:38.765282 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:35:38.768617 (+ 3335us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:35:38.768805 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:35:38.770046 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:35:38.791006 (+ 20960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:35:38.792253 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:35:38.796463 (+ 4210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:35:38.796650 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0810 22:35:38.796712 (+ 62us) server_negotiation.cc:299] Negotiation successful
0810 22:35:38.799736 (+ 3024us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20648,"thread_start_us":108,"threads_started":1}
I20260810 22:35:38.803869 30263 tablet_service.cc:1467] Processing CreateTablet for tablet f63d88532134472d9b468345f07f24b4 (DEFAULT_TABLE table=test-table [id=e7e1a0177ba045e187b6e7c07de4f1fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:35:38.805411 30263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f63d88532134472d9b468345f07f24b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:35:38.813445 30517 tablet_bootstrap.cc:492] T f63d88532134472d9b468345f07f24b4 P 39404e54bda54b45a85caac3d9f5b935: Bootstrap starting.
I20260810 22:35:38.817080 30518 tablet_bootstrap.cc:492] T f63d88532134472d9b468345f07f24b4 P 85ffe7674ce641d5ad5df9cda594e5cf: Bootstrap starting.
I20260810 22:35:38.817160 30517 tablet_bootstrap.cc:654] T f63d88532134472d9b468345f07f24b4 P 39404e54bda54b45a85caac3d9f5b935: Neither blocks nor log segments found. Creating new log.
I20260810 22:35:38.820029 30517 log.cc:824] T f63d88532134472d9b468345f07f24b4 P 39404e54bda54b45a85caac3d9f5b935: Log is configured to *not* fsync() on all Append() calls
I20260810 22:35:38.820575 30518 tablet_bootstrap.cc:654] T f63d88532134472d9b468345f07f24b4 P 85ffe7674ce641d5ad5df9cda594e5cf: Neither blocks nor log segments found. Creating new log.
I20260810 22:35:38.822301 30518 log.cc:824] T f63d88532134472d9b468345f07f24b4 P 85ffe7674ce641d5ad5df9cda594e5cf: Log is configured to *not* fsync() on all Append() calls
I20260810 22:35:38.823076 30517 tablet_bootstrap.cc:492] T f63d88532134472d9b468345f07f24b4 P 39404e54bda54b45a85caac3d9f5b935: No bootstrap required, opened a new log
I20260810 22:35:38.823298 30517 ts_tablet_manager.cc:1397] T f63d88532134472d9b468345f07f24b4 P 39404e54bda54b45a85caac3d9f5b935: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260810 22:35:38.825026 30518 tablet_bootstrap.cc:492] T f63d88532134472d9b468345f07f24b4 P 85ffe7674ce641d5ad5df9cda594e5cf: No bootstrap required, opened a new log
I20260810 22:35:38.825232 30518 ts_tablet_manager.cc:1397] T f63d88532134472d9b468345f07f24b4 P 85ffe7674ce641d5ad5df9cda594e5cf: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260810 22:35:38.831215 30517 raft_consensus.cc:348] T f63d88532134472d9b468345f07f24b4 P 39404e54bda54b45a85caac3d9f5b935 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7497bb27dae4d708948b3e78421e576" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 35091 } } peers { permanent_uuid: "85ffe7674ce641d5ad5df9cda594e5cf" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 46035 } } peers { permanent_uuid: "39404e54bda54b45a85caac3d9f5b935" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 44039 } }
I20260810 22:35:38.831576 30517 raft_consensus.cc:374] T f63d88532134472d9b468345f07f24b4 P 39404e54bda54b45a85caac3d9f5b935 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:35:38.831635 30517 raft_consensus.cc:729] T f63d88532134472d9b468345f07f24b4 P 39404e54bda54b45a85caac3d9f5b935 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39404e54bda54b45a85caac3d9f5b935, State: Initialized, Role: FOLLOWER
I20260810 22:35:38.832136 30517 consensus_queue.cc:260] T f63d88532134472d9b468345f07f24b4 P 39404e54bda54b45a85caac3d9f5b935 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7497bb27dae4d708948b3e78421e576" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 35091 } } peers { permanent_uuid: "85ffe7674ce641d5ad5df9cda594e5cf" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 46035 } } peers { permanent_uuid: "39404e54bda54b45a85caac3d9f5b935" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 44039 } }
I20260810 22:35:38.832204 30518 raft_consensus.cc:348] T f63d88532134472d9b468345f07f24b4 P 85ffe7674ce641d5ad5df9cda594e5cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7497bb27dae4d708948b3e78421e576" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 35091 } } peers { permanent_uuid: "85ffe7674ce641d5ad5df9cda594e5cf" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 46035 } } peers { permanent_uuid: "39404e54bda54b45a85caac3d9f5b935" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 44039 } }
I20260810 22:35:38.832690 30518 raft_consensus.cc:374] T f63d88532134472d9b468345f07f24b4 P 85ffe7674ce641d5ad5df9cda594e5cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:35:38.832885 30518 raft_consensus.cc:729] T f63d88532134472d9b468345f07f24b4 P 85ffe7674ce641d5ad5df9cda594e5cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85ffe7674ce641d5ad5df9cda594e5cf, State: Initialized, Role: FOLLOWER
I20260810 22:35:38.833400 30518 consensus_queue.cc:260] T f63d88532134472d9b468345f07f24b4 P 85ffe7674ce641d5ad5df9cda594e5cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7497bb27dae4d708948b3e78421e576" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 35091 } } peers { permanent_uuid: "85ffe7674ce641d5ad5df9cda594e5cf" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 46035 } } peers { permanent_uuid: "39404e54bda54b45a85caac3d9f5b935" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 44039 } }
I20260810 22:35:38.834802 30518 ts_tablet_manager.cc:1428] T f63d88532134472d9b468345f07f24b4 P 85ffe7674ce641d5ad5df9cda594e5cf: Time spent starting tablet: real 0.010s user 0.004s sys 0.004s
I20260810 22:35:38.834928 30517 ts_tablet_manager.cc:1428] T f63d88532134472d9b468345f07f24b4 P 39404e54bda54b45a85caac3d9f5b935: Time spent starting tablet: real 0.012s user 0.005s sys 0.005s
I20260810 22:35:38.835206 30338 heartbeater.cc:503] Master 127.16.233.126:44051 was elected leader, sending a full tablet report...
W20260810 22:35:38.903932 30495 tablet.cc:2369] T f63d88532134472d9b468345f07f24b4 P e7497bb27dae4d708948b3e78421e576: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 22:35:38.940790 30339 tablet.cc:2369] T f63d88532134472d9b468345f07f24b4 P 85ffe7674ce641d5ad5df9cda594e5cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 22:35:38.992342 30180 tablet.cc:2369] T f63d88532134472d9b468345f07f24b4 P 39404e54bda54b45a85caac3d9f5b935: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 10 22:35:39 dist-test-slave-77v9 krb5kdc[29371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401337, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.126@KRBTEST.COM for HTTP/127.16.233.84@KRBTEST.COM
I20260810 22:35:39.140494 30523 raft_consensus.cc:482] T f63d88532134472d9b468345f07f24b4 P 85ffe7674ce641d5ad5df9cda594e5cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:35:39.140872 30523 raft_consensus.cc:504] T f63d88532134472d9b468345f07f24b4 P 85ffe7674ce641d5ad5df9cda594e5cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7497bb27dae4d708948b3e78421e576" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 35091 } } peers { permanent_uuid: "85ffe7674ce641d5ad5df9cda594e5cf" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 46035 } } peers { permanent_uuid: "39404e54bda54b45a85caac3d9f5b935" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 44039 } }
I20260810 22:35:39.142481 30523 leader_election.cc:288] T f63d88532134472d9b468345f07f24b4 P 85ffe7674ce641d5ad5df9cda594e5cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7497bb27dae4d708948b3e78421e576 (127.16.233.67:35091), 39404e54bda54b45a85caac3d9f5b935 (127.16.233.65:44039)
I20260810 22:35:39.151503 30522 raft_consensus.cc:482] T f63d88532134472d9b468345f07f24b4 P 39404e54bda54b45a85caac3d9f5b935 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:35:39.151750 30522 raft_consensus.cc:504] T f63d88532134472d9b468345f07f24b4 P 39404e54bda54b45a85caac3d9f5b935 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7497bb27dae4d708948b3e78421e576" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 35091 } } peers { permanent_uuid: "85ffe7674ce641d5ad5df9cda594e5cf" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 46035 } } peers { permanent_uuid: "39404e54bda54b45a85caac3d9f5b935" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 44039 } }
I20260810 22:35:39.153195 30522 leader_election.cc:288] T f63d88532134472d9b468345f07f24b4 P 39404e54bda54b45a85caac3d9f5b935 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7497bb27dae4d708948b3e78421e576 (127.16.233.67:35091), 85ffe7674ce641d5ad5df9cda594e5cf (127.16.233.66:46035)
I20260810 22:35:39.158749 30509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:35:39.143054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:49019 (local address 127.16.233.67:35091)
0810 22:35:39.143187 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 22:35:39.143190 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:35:39.144127 (+ 937us) server_negotiation.cc:407] Connection header received
0810 22:35:39.144366 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:35:39.144370 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:35:39.144432 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:35:39.144524 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:35:39.150325 (+ 5801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:35:39.151098 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:35:39.157188 (+ 6090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:35:39.157828 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:35:39.157930 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:35:39.157984 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 22:35:39.158520 (+ 536us) server_negotiation.cc:299] Negotiation successful
0810 22:35:39.158613 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 22:35:39.159754 30526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:35:39.143020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.67:35091 (local address 127.16.233.66:49019)
0810 22:35:39.143993 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0810 22:35:39.144008 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:35:39.144215 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:35:39.144555 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:35:39.144558 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:35:39.144580 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:35:39.150134 (+ 5554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:35:39.150149 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:35:39.155308 (+ 5159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:35:39.155312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:35:39.157013 (+ 1701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:35:39.157027 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:35:39.158184 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:35:39.158188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:35:39.158357 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:35:39.158383 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 22:35:39.159318 (+ 935us) client_negotiation.cc:240] Negotiation successful
0810 22:35:39.159432 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":890,"thread_start_us":176,"threads_started":1}
I20260810 22:35:39.160830 30439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63d88532134472d9b468345f07f24b4" candidate_uuid: "85ffe7674ce641d5ad5df9cda594e5cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7497bb27dae4d708948b3e78421e576" is_pre_election: true
I20260810 22:35:39.161283 30439 raft_consensus.cc:2454] T f63d88532134472d9b468345f07f24b4 P e7497bb27dae4d708948b3e78421e576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85ffe7674ce641d5ad5df9cda594e5cf in term 0.
I20260810 22:35:39.162257 30210 leader_election.cc:302] T f63d88532134472d9b468345f07f24b4 P 85ffe7674ce641d5ad5df9cda594e5cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85ffe7674ce641d5ad5df9cda594e5cf, e7497bb27dae4d708948b3e78421e576; no voters:
I20260810 22:35:39.162881 30523 raft_consensus.cc:2781] T f63d88532134472d9b468345f07f24b4 P 85ffe7674ce641d5ad5df9cda594e5cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:35:39.163031 30523 raft_consensus.cc:482] T f63d88532134472d9b468345f07f24b4 P 85ffe7674ce641d5ad5df9cda594e5cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:35:39.163103 30523 raft_consensus.cc:3037] T f63d88532134472d9b468345f07f24b4 P 85ffe7674ce641d5ad5df9cda594e5cf [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:35:39.165270 30523 raft_consensus.cc:504] T f63d88532134472d9b468345f07f24b4 P 85ffe7674ce641d5ad5df9cda594e5cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7497bb27dae4d708948b3e78421e576" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 35091 } } peers { permanent_uuid: "85ffe7674ce641d5ad5df9cda594e5cf" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 46035 } } peers { permanent_uuid: "39404e54bda54b45a85caac3d9f5b935" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 44039 } }
I20260810 22:35:39.165673 30523 leader_election.cc:288] T f63d88532134472d9b468345f07f24b4 P 85ffe7674ce641d5ad5df9cda594e5cf [CANDIDATE]: Term 1 election: Requested vote from peers e7497bb27dae4d708948b3e78421e576 (127.16.233.67:35091), 39404e54bda54b45a85caac3d9f5b935 (127.16.233.65:44039)
I20260810 22:35:39.166319 30439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63d88532134472d9b468345f07f24b4" candidate_uuid: "85ffe7674ce641d5ad5df9cda594e5cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7497bb27dae4d708948b3e78421e576"
I20260810 22:35:39.166530 30439 raft_consensus.cc:3037] T f63d88532134472d9b468345f07f24b4 P e7497bb27dae4d708948b3e78421e576 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:35:39.168967 30439 raft_consensus.cc:2454] T f63d88532134472d9b468345f07f24b4 P e7497bb27dae4d708948b3e78421e576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85ffe7674ce641d5ad5df9cda594e5cf in term 1.
I20260810 22:35:39.169409 30210 leader_election.cc:302] T f63d88532134472d9b468345f07f24b4 P 85ffe7674ce641d5ad5df9cda594e5cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85ffe7674ce641d5ad5df9cda594e5cf, e7497bb27dae4d708948b3e78421e576; no voters:
I20260810 22:35:39.169670 30523 raft_consensus.cc:2781] T f63d88532134472d9b468345f07f24b4 P 85ffe7674ce641d5ad5df9cda594e5cf [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:35:39.170127 30523 raft_consensus.cc:686] T f63d88532134472d9b468345f07f24b4 P 85ffe7674ce641d5ad5df9cda594e5cf [term 1 LEADER]: Becoming Leader. State: Replica: 85ffe7674ce641d5ad5df9cda594e5cf, State: Running, Role: LEADER
I20260810 22:35:39.170536 30523 consensus_queue.cc:237] T f63d88532134472d9b468345f07f24b4 P 85ffe7674ce641d5ad5df9cda594e5cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7497bb27dae4d708948b3e78421e576" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 35091 } } peers { permanent_uuid: "85ffe7674ce641d5ad5df9cda594e5cf" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 46035 } } peers { permanent_uuid: "39404e54bda54b45a85caac3d9f5b935" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 44039 } }
I20260810 22:35:39.170995 30509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:35:39.154678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:40497 (local address 127.16.233.67:35091)
0810 22:35:39.158934 (+ 4256us) server_negotiation.cc:206] Beginning negotiation
0810 22:35:39.158938 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:35:39.160937 (+ 1999us) server_negotiation.cc:407] Connection header received
0810 22:35:39.161300 (+ 363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:35:39.161304 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:35:39.161389 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:35:39.161472 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:35:39.166602 (+ 5130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:35:39.167319 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:35:39.169597 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:35:39.170178 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:35:39.170271 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:35:39.170333 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 22:35:39.170780 (+ 447us) server_negotiation.cc:299] Negotiation successful
0810 22:35:39.170861 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4195,"thread_start_us":124,"threads_started":1}
I20260810 22:35:39.171675 30529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:35:39.153561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.67:35091 (local address 127.16.233.65:40497)
0810 22:35:39.160765 (+ 7204us) negotiation.cc:107] Waiting for socket to connect
0810 22:35:39.160791 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0810 22:35:39.161068 (+ 277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:35:39.166094 (+ 5026us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:35:39.166098 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:35:39.166120 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:35:39.166436 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:35:39.166444 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:35:39.167469 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:35:39.167472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:35:39.169451 (+ 1979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:35:39.169463 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:35:39.170509 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:35:39.170512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:35:39.170633 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:35:39.170656 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 22:35:39.171196 (+ 540us) client_negotiation.cc:240] Negotiation successful
0810 22:35:39.171288 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7078,"mutex_wait_us":103,"thread_start_us":193,"threads_started":1}
I20260810 22:35:39.172309 30437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63d88532134472d9b468345f07f24b4" candidate_uuid: "39404e54bda54b45a85caac3d9f5b935" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7497bb27dae4d708948b3e78421e576" is_pre_election: true
I20260810 22:35:39.172551 30437 raft_consensus.cc:2379] T f63d88532134472d9b468345f07f24b4 P e7497bb27dae4d708948b3e78421e576 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39404e54bda54b45a85caac3d9f5b935 in current term 1: Already voted for candidate 85ffe7674ce641d5ad5df9cda594e5cf in this term.
I20260810 22:35:39.172920 30511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:35:39.154991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:40807 (local address 127.16.233.66:46035)
0810 22:35:39.155134 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 22:35:39.155139 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:35:39.161576 (+ 6437us) server_negotiation.cc:407] Connection header received
0810 22:35:39.165108 (+ 3532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:35:39.165112 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:35:39.165163 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:35:39.165214 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:35:39.166254 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:35:39.167187 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:35:39.171433 (+ 4246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:35:39.172085 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:35:39.172183 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:35:39.172228 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0810 22:35:39.172699 (+ 471us) server_negotiation.cc:299] Negotiation successful
0810 22:35:39.172790 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 22:35:39.173615 30527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:35:39.154075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.66:46035 (local address 127.16.233.65:40807)
0810 22:35:39.161429 (+ 7354us) negotiation.cc:107] Waiting for socket to connect
0810 22:35:39.161446 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 22:35:39.164936 (+ 3490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:35:39.165376 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:35:39.165379 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:35:39.165400 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:35:39.166082 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:35:39.166094 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:35:39.167477 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:35:39.167481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:35:39.169625 (+ 2144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:35:39.169637 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:35:39.172345 (+ 2708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:35:39.172348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:35:39.172522 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:35:39.172554 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 22:35:39.173247 (+ 693us) client_negotiation.cc:240] Negotiation successful
0810 22:35:39.173335 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7269,"mutex_wait_us":3,"thread_start_us":159,"threads_started":1}
I20260810 22:35:39.174340 29938 catalog_manager.cc:5495] T f63d88532134472d9b468345f07f24b4 P 85ffe7674ce641d5ad5df9cda594e5cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 85ffe7674ce641d5ad5df9cda594e5cf (127.16.233.66). New cstate: current_term: 1 leader_uuid: "85ffe7674ce641d5ad5df9cda594e5cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7497bb27dae4d708948b3e78421e576" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 35091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85ffe7674ce641d5ad5df9cda594e5cf" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 46035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39404e54bda54b45a85caac3d9f5b935" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 44039 } health_report { overall_health: UNKNOWN } } }
I20260810 22:35:39.174685 30265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63d88532134472d9b468345f07f24b4" candidate_uuid: "39404e54bda54b45a85caac3d9f5b935" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85ffe7674ce641d5ad5df9cda594e5cf" is_pre_election: true
I20260810 22:35:39.176353 30049 leader_election.cc:302] T f63d88532134472d9b468345f07f24b4 P 39404e54bda54b45a85caac3d9f5b935 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39404e54bda54b45a85caac3d9f5b935; no voters: 85ffe7674ce641d5ad5df9cda594e5cf, e7497bb27dae4d708948b3e78421e576
I20260810 22:35:39.177016 30522 raft_consensus.cc:3037] T f63d88532134472d9b468345f07f24b4 P 39404e54bda54b45a85caac3d9f5b935 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:35:39.178025 30513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:35:39.143927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:34347 (local address 127.16.233.65:44039)
0810 22:35:39.144074 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 22:35:39.144079 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:35:39.145808 (+ 1729us) server_negotiation.cc:407] Connection header received
0810 22:35:39.145989 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:35:39.145993 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:35:39.146049 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:35:39.146155 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:35:39.146556 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:35:39.147119 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:35:39.176557 (+ 29438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:35:39.177150 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:35:39.177268 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:35:39.177317 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 22:35:39.177829 (+ 512us) server_negotiation.cc:299] Negotiation successful
0810 22:35:39.177911 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 22:35:39.178517 30525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:35:39.143534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.65:44039 (local address 127.16.233.66:34347)
0810 22:35:39.145681 (+ 2147us) negotiation.cc:107] Waiting for socket to connect
0810 22:35:39.145695 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:35:39.145841 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:35:39.146166 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:35:39.146168 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:35:39.146185 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:35:39.146414 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:35:39.146421 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:35:39.147238 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:35:39.147243 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:35:39.176423 (+ 29180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:35:39.176436 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:35:39.177383 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:35:39.177387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:35:39.177650 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:35:39.177677 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 22:35:39.178211 (+ 534us) client_negotiation.cc:240] Negotiation successful
0810 22:35:39.178302 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2084,"thread_start_us":173,"threads_started":1}
I20260810 22:35:39.178936 30522 raft_consensus.cc:2726] T f63d88532134472d9b468345f07f24b4 P 39404e54bda54b45a85caac3d9f5b935 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 22:35:39.180415 30124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63d88532134472d9b468345f07f24b4" candidate_uuid: "85ffe7674ce641d5ad5df9cda594e5cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39404e54bda54b45a85caac3d9f5b935" is_pre_election: true
I20260810 22:35:39.180641 30124 raft_consensus.cc:2454] T f63d88532134472d9b468345f07f24b4 P 39404e54bda54b45a85caac3d9f5b935 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85ffe7674ce641d5ad5df9cda594e5cf in term 1.
I20260810 22:35:39.181048 30123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63d88532134472d9b468345f07f24b4" candidate_uuid: "85ffe7674ce641d5ad5df9cda594e5cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39404e54bda54b45a85caac3d9f5b935"
I20260810 22:35:39.182861 30123 raft_consensus.cc:2454] T f63d88532134472d9b468345f07f24b4 P 39404e54bda54b45a85caac3d9f5b935 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85ffe7674ce641d5ad5df9cda594e5cf in term 1.
I20260810 22:35:39.218672 30511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:35:39.213600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48048 (local address 127.16.233.66:46035)
0810 22:35:39.213875 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0810 22:35:39.213881 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:35:39.213899 (+ 18us) server_negotiation.cc:407] Connection header received
0810 22:35:39.213994 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:35:39.213998 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:35:39.214072 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:35:39.214202 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:35:39.214485 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:35:39.215244 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:35:39.216471 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:35:39.216797 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:35:39.216918 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:35:39.217351 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:35:39.217970 (+ 619us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:35:39.218087 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 22:35:39.218471 (+ 384us) server_negotiation.cc:299] Negotiation successful
0810 22:35:39.218527 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163}
I20260810 22:35:39.230621 30437 raft_consensus.cc:1261] T f63d88532134472d9b468345f07f24b4 P e7497bb27dae4d708948b3e78421e576 [term 1 FOLLOWER]: Refusing update from remote peer 85ffe7674ce641d5ad5df9cda594e5cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:35:39.231339 30124 raft_consensus.cc:1261] T f63d88532134472d9b468345f07f24b4 P 39404e54bda54b45a85caac3d9f5b935 [term 1 FOLLOWER]: Refusing update from remote peer 85ffe7674ce641d5ad5df9cda594e5cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:35:39.231423 30531 consensus_queue.cc:1035] T f63d88532134472d9b468345f07f24b4 P 85ffe7674ce641d5ad5df9cda594e5cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7497bb27dae4d708948b3e78421e576" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 35091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:35:39.233258 30531 consensus_queue.cc:1035] T f63d88532134472d9b468345f07f24b4 P 85ffe7674ce641d5ad5df9cda594e5cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "39404e54bda54b45a85caac3d9f5b935" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 44039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:35:39.262269 30536 mvcc.cc:205] Tried to move back new op lower bound from 7317099885475999744 to 7317099885245865984. Current Snapshot: MvccSnapshot[applied={T|T < 7317099885475999744}]
I20260810 22:35:39.267591 30539 mvcc.cc:205] Tried to move back new op lower bound from 7317099885475999744 to 7317099885245865984. Current Snapshot: MvccSnapshot[applied={T|T < 7317099885475999744}]
I20260810 22:35:39.269698 30537 mvcc.cc:205] Tried to move back new op lower bound from 7317099885475999744 to 7317099885245865984. Current Snapshot: MvccSnapshot[applied={T|T < 7317099885475999744}]
I20260810 22:35:39.278430 29938 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:35:39.278635 29938 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:35:39.281584 29938 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6fb71a43b486495f810001168e757d58: Sending DeleteTablet for 3 replicas of tablet f63d88532134472d9b468345f07f24b4
I20260810 22:35:39.282712 30101 tablet_service.cc:1514] Processing DeleteTablet for tablet f63d88532134472d9b468345f07f24b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:35:39 UTC) from {username='kudu', principal='kudu/127.16.233.126@KRBTEST.COM'} at 127.0.0.1:43698
I20260810 22:35:39.283643 30263 tablet_service.cc:1514] Processing DeleteTablet for tablet f63d88532134472d9b468345f07f24b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:35:39 UTC) from {username='kudu', principal='kudu/127.16.233.126@KRBTEST.COM'} at 127.0.0.1:60750
I20260810 22:35:39.284998 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 30030
I20260810 22:35:39.285291 30410 tablet_service.cc:1514] Processing DeleteTablet for tablet f63d88532134472d9b468345f07f24b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:35:39 UTC) from {username='kudu', principal='kudu/127.16.233.126@KRBTEST.COM'} at 127.0.0.1:50138
I20260810 22:35:39.295295 30548 tablet_replica.cc:330] T f63d88532134472d9b468345f07f24b4 P e7497bb27dae4d708948b3e78421e576: stopping tablet replica
I20260810 22:35:39.295780 30548 raft_consensus.cc:2229] T f63d88532134472d9b468345f07f24b4 P e7497bb27dae4d708948b3e78421e576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:35:39.296193 30548 raft_consensus.cc:2258] T f63d88532134472d9b468345f07f24b4 P e7497bb27dae4d708948b3e78421e576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:35:39.299203 30547 tablet_replica.cc:330] T f63d88532134472d9b468345f07f24b4 P 85ffe7674ce641d5ad5df9cda594e5cf: stopping tablet replica
I20260810 22:35:39.299917 30548 ts_tablet_manager.cc:1905] T f63d88532134472d9b468345f07f24b4 P e7497bb27dae4d708948b3e78421e576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:35:39.300364 30547 raft_consensus.cc:2229] T f63d88532134472d9b468345f07f24b4 P 85ffe7674ce641d5ad5df9cda594e5cf [term 1 LEADER]: Raft consensus shutting down.
I20260810 22:35:39.304139 30547 raft_consensus.cc:2258] T f63d88532134472d9b468345f07f24b4 P 85ffe7674ce641d5ad5df9cda594e5cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:35:39.308630 30548 ts_tablet_manager.cc:1918] T f63d88532134472d9b468345f07f24b4 P e7497bb27dae4d708948b3e78421e576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:35:39.312561 30547 ts_tablet_manager.cc:1905] T f63d88532134472d9b468345f07f24b4 P 85ffe7674ce641d5ad5df9cda594e5cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:35:39.316289 30547 ts_tablet_manager.cc:1918] T f63d88532134472d9b468345f07f24b4 P 85ffe7674ce641d5ad5df9cda594e5cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:35:39.316627 30548 log.cc:1196] T f63d88532134472d9b468345f07f24b4 P e7497bb27dae4d708948b3e78421e576: Deleting WAL directory at /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/ts-2/wal/wals/f63d88532134472d9b468345f07f24b4
I20260810 22:35:39.317236 30548 ts_tablet_manager.cc:1939] T f63d88532134472d9b468345f07f24b4 P e7497bb27dae4d708948b3e78421e576: Deleting consensus metadata
W20260810 22:35:39.318389 29923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.233.65:44039 (error 108)
I20260810 22:35:39.321364 29922 catalog_manager.cc:4841] TS e7497bb27dae4d708948b3e78421e576 (127.16.233.67:35091): tablet f63d88532134472d9b468345f07f24b4 (table test-table [id=e7e1a0177ba045e187b6e7c07de4f1fa]) successfully deleted
I20260810 22:35:39.321473 30547 log.cc:1196] T f63d88532134472d9b468345f07f24b4 P 85ffe7674ce641d5ad5df9cda594e5cf: Deleting WAL directory at /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786401215402979-17317-0/minicluster-data/ts-1/wal/wals/f63d88532134472d9b468345f07f24b4
I20260810 22:35:39.321761 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 30189
I20260810 22:35:39.327464 30550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:35:39.326710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.65:44039 (local address 127.0.0.1:38194)
0810 22:35:39.327192 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0810 22:35:39.327297 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.65:44039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":83,"threads_started":1}
W20260810 22:35:39.327916 29923 catalog_manager.cc:4584] TS 39404e54bda54b45a85caac3d9f5b935 (127.16.233.65:44039): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f63d88532134472d9b468345f07f24b4: Network error: Client connection negotiation failed: client connection to 127.16.233.65:44039: connect: Connection refused (error 111)
W20260810 22:35:39.335449 30550 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:35:39.334616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.66:46035 (local address 127.0.0.1:48062)
0810 22:35:39.334847 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0810 22:35:39.334867 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 22:35:39.334997 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:35:39.335300 (+ 303us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.66:46035: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":137}
I20260810 22:35:39.336436 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 30345
I20260810 22:35:39.357033 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 29903
I20260810 22:35:39.371479 17317 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260810 22:35:39.717383 17317 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260810 22:35:39.717522 17317 mini_ranger.cc:66] Stopping Ranger...
I20260810 22:35:40.063195 17317 mini_ranger.cc:68] Stopped Ranger
2026-08-10 22:35:40.063 UTC [29405] LOG: received smart shutdown request
2026-08-10 22:35:40.068 UTC [29405] LOG: background worker "logical replication launcher" (PID 29416) exited with exit code 1
2026-08-10 22:35:40.068 UTC [29410] LOG: shutting down
2026-08-10 22:35:40.257 UTC [29405] LOG: database system is shut down
2026-08-10T22:35:40Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (55093 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:35:40 dist-test-slave-77v9 krb5kdc[30565](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:35:40 dist-test-slave-77v9 krb5kdc[30565](info): set up 2 sockets
Aug 10 22:35:40 dist-test-slave-77v9 krb5kdc[30565](info): commencing operation
krb5kdc: starting...
W20260810 22:35:41.587669 17317 mini_kdc.cc:121] Time spent starting KDC: real 1.158s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Aug 10 22:35:41 dist-test-slave-77v9 krb5kdc[30565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401341, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:35:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:35:41Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.16.233.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.16.233.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.16.233.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/krb5kdc/rangeradmin_127.16.233.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.16.233.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/krb5kdc/rangeradmin_127.16.233.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.16.233.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.16.233.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.16.233.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/krb5kdc/rangerlookup_127.16.233.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.16.233.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/krb5kdc/rangerlookup_127.16.233.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.233.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.16.233.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.233.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/krb5kdc/HTTP_127.16.233.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.233.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/krb5kdc/HTTP_127.16.233.84@KRBTEST.COM.keytab.
I20260810 22:35:41.801497 17317 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskfcURSo/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/postgres -l logfile start
2026-08-10 22:35:44.669 UTC [30605] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-10 22:35:44.669 UTC [30605] LOG: listening on IPv4 address "127.16.233.84", port 40045
2026-08-10 22:35:44.672 UTC [30605] LOG: listening on Unix socket "/tmp/.s.PGSQL.40045"
2026-08-10 22:35:44.695 UTC [30608] LOG: database system was shut down at 2026-08-10 22:35:42 UTC
2026-08-10 22:35:44.700 UTC [30605] LOG: database system is ready to accept connections
I20260810 22:35:45.731320 17317 mini_postgres.cc:96] Postgres bound to 40045
2026-08-10 22:35:45.739 UTC [30616] FATAL: database "slave" does not exist
127.16.233.84:40045 - accepting connections
I20260810 22:35:45.739845 17317 mini_ranger.cc:161] Starting Ranger...
I20260810 22:35:45.756919 17317 mini_ranger.cc:84] Created miniranger Postgres user
I20260810 22:35:46.358139 17317 mini_ranger.cc:87] Created ranger Postgres database
I20260810 22:35:46.358229 17317 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ranger-admin
2026-08-10 22:35:46,642 [I] DB FLAVOR :POSTGRES
2026-08-10 22:35:46,642 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 22:35:46,642 [I] Checking connection..
2026-08-10 22:35:46,642 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfcURSo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfcURSo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.84:40045/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-10 22:35:46,929 [I] Checking connection passed.
2026-08-10 22:35:46,929 [I] --------- Verifying version history table ---------
2026-08-10 22:35:46,929 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfcURSo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfcURSo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.84:40045/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 22:35:47,194 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 22:35:47,194 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfcURSo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfcURSo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.84:40045/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 22:35:47,484 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 22:35:47,485 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-10 22:35:47,485 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfcURSo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfcURSo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.84:40045/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskfcURSo/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-10 22:35:47.769 UTC [30696] WARNING: there is no transaction in progress
2026-08-10 22:35:47,778 [I] create_dbversion_catalog.sql file imported successfully
2026-08-10 22:35:47,778 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfcURSo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfcURSo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.84:40045/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 22:35:48,061 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-10 22:35:48,061 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 22:35:48,061 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfcURSo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfcURSo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.84:40045/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-10 22:35:48,354 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfcURSo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfcURSo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.84:40045/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-10 22:35:48,620 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfcURSo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfcURSo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.84:40045/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-77v9.c.gcp-upstream.internal','N') ;"
2026-08-10 22:35:48,888 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfcURSo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfcURSo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.84:40045/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 22:35:49,169 [I] Table x_portal_user does not exist in database ranger
2026-08-10 22:35:49,169 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfcURSo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfcURSo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.84:40045/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 22:35:49,446 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-10 22:35:49,446 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfcURSo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfcURSo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.84:40045/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 22:35:49,731 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfcURSo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfcURSo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.84:40045/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 22:35:50,013 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-10 22:35:50,014 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfcURSo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfcURSo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.84:40045/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskfcURSo/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-10 22:35:51.146 UTC [30822] WARNING: there is no transaction in progress
2026-08-10 22:35:51.165 UTC [30822] WARNING: there is no transaction in progress
2026-08-10 22:35:51.186 UTC [30822] WARNING: there is no transaction in progress
2026-08-10 22:35:51.204 UTC [30822] WARNING: there is no transaction in progress
2026-08-10 22:35:51.226 UTC [30822] WARNING: there is no transaction in progress
2026-08-10 22:35:51.250 UTC [30822] WARNING: there is no transaction in progress
2026-08-10 22:35:51.318 UTC [30822] WARNING: there is no transaction in progress
2026-08-10 22:35:51.326 UTC [30822] WARNING: there is no transaction in progress
2026-08-10 22:35:51.344 UTC [30822] WARNING: there is no transaction in progress
2026-08-10 22:35:51.357 UTC [30822] WARNING: there is no transaction in progress
2026-08-10 22:35:51.370 UTC [30822] WARNING: there is no transaction in progress
2026-08-10 22:35:51.388 UTC [30822] WARNING: there is no transaction in progress
2026-08-10 22:35:51.401 UTC [30822] WARNING: there is no transaction in progress
2026-08-10 22:35:51.410 UTC [30822] WARNING: there is no transaction in progress
2026-08-10 22:35:51.420 UTC [30822] WARNING: there is no transaction in progress
2026-08-10 22:35:51.429 UTC [30822] WARNING: there is no transaction in progress
2026-08-10 22:35:52,272 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-10 22:35:52,273 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfcURSo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfcURSo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.84:40045/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-10 22:35:52,539 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-10 22:35:52,539 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfcURSo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfcURSo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.84:40045/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 22:35:52,822 [I] Table x_portal_user already exists in database 'ranger'
2026-08-10 22:35:52,822 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfcURSo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfcURSo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.84:40045/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 22:35:53,122 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-10 22:35:53,123 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfcURSo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfcURSo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.84:40045/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 22:35:53,399 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfcURSo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfcURSo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.84:40045/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 22:35:53,705 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfcURSo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfcURSo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.84:40045/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-77v9.c.gcp-upstream.internal';"
2026-08-10 22:35:53,982 [I] CORE_DB_SCHEMA import status has been updated
2026-08-10 22:35:53,982 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfcURSo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfcURSo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.84:40045/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-10 22:35:54,286 [I] DB_PATCHES have already been applied
I20260810 22:35:54.293183 17317 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ranger-admin:/tmp/dist-test-taskfcURSo/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskfcURSo/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskfcURSo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfcURSo/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260810 22:35:54.293290 17317 mini_ranger.cc:193] Using host: 127.16.233.84
I20260810 22:35:54.298275 17317 mini_ranger.cc:227] Ranger admin URL: http://127.16.233.84:53359
Aug 10, 2026 10:35:54 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 10:35:54 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 10:35:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ranger-admin/ews/webapp
Aug 10, 2026 10:35:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 10, 2026 10:35:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 10, 2026 10:35:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ranger-admin/ews/webapp].
Aug 10, 2026 10:35:55 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260810 22:35:55.569335 17317 mini_ranger.cc:160] Time spent starting Ranger: real 9.829s user 0.003s sys 0.010s
Aug 10, 2026 10:36:16 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 10, 2026 10:36:16 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Aug 10, 2026 10:36:16 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 10, 2026 10:36:17 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 10, 2026 10:36:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 10, 2026 10:36:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 10, 2026 10:36:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 10, 2026 10:36:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 10, 2026 10:36:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 10, 2026 10:36:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 10, 2026 10:36:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 10, 2026 10:36:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 10, 2026 10:36:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 10, 2026 10:36:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 10, 2026 10:36:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 10, 2026 10:36:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 10, 2026 10:36:17 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 10, 2026 10:36:18 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260810 22:36:18.696571 17317 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.16.233.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.16.233.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.16.233.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/krb5kdc/rangerkms_127.16.233.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.16.233.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/krb5kdc/rangerkms_127.16.233.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.233.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.16.233.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.233.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/krb5kdc/HTTP_127.16.233.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.233.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/krb5kdc/HTTP_127.16.233.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 10 22:36:18 dist-test-slave-77v9 krb5kdc[30565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401378, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260810 22:36:18.804924 17317 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260810 22:36:18.817219 17317 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260810 22:36:19.909847 17317 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260810 22:36:19.909966 17317 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ranger-kms
I20260810 22:36:19.909991 17317 mini_ranger_kms.cc:227] Using postgres at 127.16.233.84:40045
2026-08-10 22:36:20,097 [I] DB FLAVOR :POSTGRES
2026-08-10 22:36:20,098 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 22:36:20,098 [I] Checking connection
2026-08-10 22:36:20,098 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfcURSo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.84:40045/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-10 22:36:20,363 [I] connection success
2026-08-10 22:36:20,364 [I] --------- Verifying Ranger DB tables ---------
2026-08-10 22:36:20,364 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-10 22:36:20,364 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfcURSo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.84:40045/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-10 22:36:20,640 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-10 22:36:20,640 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 22:36:20,641 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-10 22:36:20,641 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfcURSo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.233.84:40045/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-10 22:36:20,969 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260810 22:36:21.565500 17317 mini_ranger_kms.cc:310] Created kms service
I20260810 22:36:21.801891 17317 mini_ranger_kms.cc:326] Created kudu user
I20260810 22:36:21.854426 17317 mini_ranger_kms.cc:343] Created rangerkms user
Aug 10 22:36:21 dist-test-slave-77v9 krb5kdc[30565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401378, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.16.233.84@KRBTEST.COM
I20260810 22:36:22.288466 17317 mini_ranger_kms.cc:384] Added ranger policy
I20260810 22:36:22.288744 17317 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ranger-kms:/tmp/dist-test-taskfcURSo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfcURSo/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskfcURSo/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260810 22:36:22.288774 17317 mini_ranger_kms.cc:240] Using host: 127.16.233.84
I20260810 22:36:22.293779 17317 mini_ranger_kms.cc:276] Ranger KMS PID: 31024
I20260810 22:36:22.293881 17317 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.16.233.84:52919
Aug 10, 2026 10:36:23 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 10:36:23 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 10:36:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 10, 2026 10:36:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 10, 2026 10:36:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ranger-kms/ews/webapp].
Aug 10, 2026 10:36:23 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260810 22:36:24.772845 17317 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.968s user 0.021s sys 0.151s
I20260810 22:36:24.772975 17317 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260810 22:36:24.773042 17317 mini_ranger_kms.cc:401] 127.16.233.84:52919/kms/v1/keys
Aug 10 22:36:24 dist-test-slave-77v9 krb5kdc[30565](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786401384, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.233.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 10 22:36:25 dist-test-slave-77v9 krb5kdc[30565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401384, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.233.84@KRBTEST.COM for HTTP/127.16.233.84@KRBTEST.COM
Aug 10 22:36:25 dist-test-slave-77v9 krb5kdc[30565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401384, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.233.84@KRBTEST.COM for HTTP/127.16.233.84@KRBTEST.COM
Aug 10 22:36:28 dist-test-slave-77v9 krb5kdc[30565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.16.233.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab.
Entry for principal kudu/127.16.233.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab.
I20260810 22:36:28.161746 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:40771
--webserver_interface=127.16.233.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab
--principal=kudu/127.16.233.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:35449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.126:40771
--ranger_config_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.233.84:52919/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:36:28.285004 31082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:36:28.285308 31082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:36:28.285351 31082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:36:28.289657 31082 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:36:28.289733 31082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:36:28.289753 31082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:36:28.289770 31082 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:36:28.289789 31082 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:36:28.295814 31082 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:35449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/master-0/wal
--ranger_kms_url=127.16.233.84:52919/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.16.233.126:40771
--ranger_config_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab
--principal=kudu/127.16.233.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:40771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31082
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:36:28.297134 31082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:36:28.298328 31082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:36:28.307708 31087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:36:28.307965 31082 server_base.cc:1034] running on GCE node
W20260810 22:36:28.307723 31088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:36:28.307723 31090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:36:28.308880 31082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:36:28.310360 31082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:36:28.311596 31082 hybrid_clock.cc:648] HybridClock initialized: now 1786401388311570 us; error 64 us; skew 500 ppm
Aug 10 22:36:28 dist-test-slave-77v9 krb5kdc[30565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401388, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:36:28.314682 31082 init.cc:377] Logged in from keytab as kudu/127.16.233.126@KRBTEST.COM (short username kudu)
I20260810 22:36:28.315824 31082 webserver.cc:460] Webserver started at http://127.16.233.126:43393/ using document root <none> and password file <none>
I20260810 22:36:28.316414 31082 fs_manager.cc:359] Metadata directory not provided
I20260810 22:36:28.316476 31082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:36:28.316713 31082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 22:36:28 dist-test-slave-77v9 krb5kdc[30565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401388, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.126@KRBTEST.COM for HTTP/127.16.233.84@KRBTEST.COM
I20260810 22:36:28.376746 31082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/master-0/data/instance:
uuid: "1176e1761df84e2b94862e38dd0a8d36"
format_stamp: "Formatted at 2026-08-10 22:36:28 on dist-test-slave-77v9"
server_key: "b69e222987b7c9eae15a2f0310247075"
server_key_iv: "10c1ecb1bdd3f0a9059c2c088d73bad8"
server_key_version: "kuduclusterkey@0"
I20260810 22:36:28.377631 31082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/master-0/wal/instance:
uuid: "1176e1761df84e2b94862e38dd0a8d36"
format_stamp: "Formatted at 2026-08-10 22:36:28 on dist-test-slave-77v9"
server_key: "b69e222987b7c9eae15a2f0310247075"
server_key_iv: "10c1ecb1bdd3f0a9059c2c088d73bad8"
server_key_version: "kuduclusterkey@0"
I20260810 22:36:28.382093 31082 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260810 22:36:28.399061 31098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:36:28.400869 31082 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:36:28.401037 31082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/master-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/master-0/wal
uuid: "1176e1761df84e2b94862e38dd0a8d36"
format_stamp: "Formatted at 2026-08-10 22:36:28 on dist-test-slave-77v9"
server_key: "b69e222987b7c9eae15a2f0310247075"
server_key_iv: "10c1ecb1bdd3f0a9059c2c088d73bad8"
server_key_version: "kuduclusterkey@0"
I20260810 22:36:28.401190 31082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:36:28.425874 31082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:36:28.429822 31082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:36:28.430043 31082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:36:28.438876 31082 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.126:40771
I20260810 22:36:28.438972 31157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.126:40771 every 8 connection(s)
I20260810 22:36:28.440129 31082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
I20260810 22:36:28.441963 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 31082
I20260810 22:36:28.442098 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/master-0/wal/instance
I20260810 22:36:28.442414 17317 external_mini_cluster.cc:1428] Setting key 9cb40803ad9de3c0cb7005293a0e5a5f
I20260810 22:36:28.443933 31158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:36:28.452046 31158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1176e1761df84e2b94862e38dd0a8d36: Bootstrap starting.
Aug 10 22:36:28 dist-test-slave-77v9 krb5kdc[30565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:36:28.455147 31158 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1176e1761df84e2b94862e38dd0a8d36: Neither blocks nor log segments found. Creating new log.
I20260810 22:36:28.456290 31158 log.cc:824] T 00000000000000000000000000000000 P 1176e1761df84e2b94862e38dd0a8d36: Log is configured to *not* fsync() on all Append() calls
I20260810 22:36:28.458989 31158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1176e1761df84e2b94862e38dd0a8d36: No bootstrap required, opened a new log
I20260810 22:36:28.462502 31164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:28.444107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38678 (local address 127.16.233.126:40771)
0810 22:36:28.444737 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0810 22:36:28.444762 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0810 22:36:28.444832 (+ 70us) server_negotiation.cc:407] Connection header received
0810 22:36:28.445800 (+ 968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:36:28.445826 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:36:28.446287 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:36:28.446754 (+ 467us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:36:28.447748 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:28.449544 (+ 1796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:28.450866 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:28.451650 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:28.451836 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:28.455482 (+ 3646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:36:28.455493 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:36:28.455516 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:36:28.455564 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:36:28.458528 (+ 2964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:36:28.459226 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:36:28.459233 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:36:28.459241 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:36:28.459398 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:36:28.459787 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:36:28.459790 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:36:28.459792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:36:28.460409 (+ 617us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:36:28.460616 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0810 22:36:28.461026 (+ 410us) server_negotiation.cc:299] Negotiation successful
0810 22:36:28.461367 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":178,"threads_started":1}
I20260810 22:36:28.463653 31158 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1176e1761df84e2b94862e38dd0a8d36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1176e1761df84e2b94862e38dd0a8d36" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 40771 } }
I20260810 22:36:28.464047 31158 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1176e1761df84e2b94862e38dd0a8d36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:36:28.464125 31158 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1176e1761df84e2b94862e38dd0a8d36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1176e1761df84e2b94862e38dd0a8d36, State: Initialized, Role: FOLLOWER
I20260810 22:36:28.464671 31158 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1176e1761df84e2b94862e38dd0a8d36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1176e1761df84e2b94862e38dd0a8d36" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 40771 } }
I20260810 22:36:28.464867 31158 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1176e1761df84e2b94862e38dd0a8d36 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:36:28.464962 31158 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1176e1761df84e2b94862e38dd0a8d36 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:36:28.465101 31158 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1176e1761df84e2b94862e38dd0a8d36 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:36:28.466272 31158 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1176e1761df84e2b94862e38dd0a8d36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1176e1761df84e2b94862e38dd0a8d36" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 40771 } }
I20260810 22:36:28.466684 31158 leader_election.cc:302] T 00000000000000000000000000000000 P 1176e1761df84e2b94862e38dd0a8d36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1176e1761df84e2b94862e38dd0a8d36; no voters:
I20260810 22:36:28.467077 31158 leader_election.cc:288] T 00000000000000000000000000000000 P 1176e1761df84e2b94862e38dd0a8d36 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:36:28.467206 31166 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1176e1761df84e2b94862e38dd0a8d36 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:36:28.467464 31166 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1176e1761df84e2b94862e38dd0a8d36 [term 1 LEADER]: Becoming Leader. State: Replica: 1176e1761df84e2b94862e38dd0a8d36, State: Running, Role: LEADER
I20260810 22:36:28.467847 31166 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1176e1761df84e2b94862e38dd0a8d36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1176e1761df84e2b94862e38dd0a8d36" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 40771 } }
I20260810 22:36:28.468597 31158 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1176e1761df84e2b94862e38dd0a8d36 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:36:28.469764 31167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1176e1761df84e2b94862e38dd0a8d36 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1176e1761df84e2b94862e38dd0a8d36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1176e1761df84e2b94862e38dd0a8d36" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 40771 } } }
I20260810 22:36:28.469946 31167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1176e1761df84e2b94862e38dd0a8d36 [sys.catalog]: This master's current role is: LEADER
I20260810 22:36:28.470453 31168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1176e1761df84e2b94862e38dd0a8d36 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1176e1761df84e2b94862e38dd0a8d36. Latest consensus state: current_term: 1 leader_uuid: "1176e1761df84e2b94862e38dd0a8d36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1176e1761df84e2b94862e38dd0a8d36" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 40771 } } }
I20260810 22:36:28.470795 31168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1176e1761df84e2b94862e38dd0a8d36 [sys.catalog]: This master's current role is: LEADER
I20260810 22:36:28.471481 31175 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:36:28.471613 31158 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 22:36:28.477720 31175 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:36:28.490166 31175 catalog_manager.cc:1347] Generated new cluster ID: f13b3238a3b34e0a84d7b302b6d6f075
I20260810 22:36:28.490262 31175 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:36:28.529680 31175 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:36:28.531138 31175 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:36:28.540526 31175 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1176e1761df84e2b94862e38dd0a8d36: Generated new TSK 0
I20260810 22:36:28.541617 31175 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.233.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab.
Entry for principal kudu/127.16.233.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab.
I20260810 22:36:29.300206 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.233.84:52919/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.65:0
--local_ip_for_outbound_sockets=127.16.233.65
--webserver_interface=127.16.233.65
--webserver_port=0
--tserver_master_addrs=127.16.233.126:40771
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab
--principal=kudu/127.16.233.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:35449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:36:29.480635 31218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:36:29.481096 31218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:36:29.481180 31218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:36:29.487519 31218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:36:29.487649 31218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:36:29.487766 31218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.65
I20260810 22:36:29.496186 31218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:35449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.16.233.84:52919/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab
--principal=kudu/127.16.233.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.233.65
--webserver_port=0
--tserver_master_addrs=127.16.233.126:40771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31218
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.65
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:36:29.497663 31218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:36:29.498942 31218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:36:29.512944 31227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:36:29.512950 31229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:36:29.512944 31226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:36:29.513111 31218 server_base.cc:1034] running on GCE node
I20260810 22:36:29.514040 31218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:36:29.514835 31218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:36:29.516067 31218 hybrid_clock.cc:648] HybridClock initialized: now 1786401389516039 us; error 52 us; skew 500 ppm
Aug 10 22:36:29 dist-test-slave-77v9 krb5kdc[30565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401389, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:36:29.519901 31218 init.cc:377] Logged in from keytab as kudu/127.16.233.65@KRBTEST.COM (short username kudu)
I20260810 22:36:29.521090 31218 webserver.cc:460] Webserver started at http://127.16.233.65:41927/ using document root <none> and password file <none>
I20260810 22:36:29.521803 31218 fs_manager.cc:359] Metadata directory not provided
I20260810 22:36:29.521903 31218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:36:29.522146 31218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 22:36:29 dist-test-slave-77v9 krb5kdc[30565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401389, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.65@KRBTEST.COM for HTTP/127.16.233.84@KRBTEST.COM
I20260810 22:36:29.542172 31218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-0/data/instance:
uuid: "fc6f41614da44d429522b576d0ff0187"
format_stamp: "Formatted at 2026-08-10 22:36:29 on dist-test-slave-77v9"
server_key: "860292d7d4f6140aed32c948b7519da0"
server_key_iv: "4644537329984c7c51a7d21b29e68386"
server_key_version: "kuduclusterkey@0"
I20260810 22:36:29.543195 31218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-0/wal/instance:
uuid: "fc6f41614da44d429522b576d0ff0187"
format_stamp: "Formatted at 2026-08-10 22:36:29 on dist-test-slave-77v9"
server_key: "860292d7d4f6140aed32c948b7519da0"
server_key_iv: "4644537329984c7c51a7d21b29e68386"
server_key_version: "kuduclusterkey@0"
I20260810 22:36:29.555317 31218 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.000s sys 0.004s
I20260810 22:36:29.578228 31237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:36:29.580219 31218 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.002s sys 0.000s
I20260810 22:36:29.580497 31218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-0/wal
uuid: "fc6f41614da44d429522b576d0ff0187"
format_stamp: "Formatted at 2026-08-10 22:36:29 on dist-test-slave-77v9"
server_key: "860292d7d4f6140aed32c948b7519da0"
server_key_iv: "4644537329984c7c51a7d21b29e68386"
server_key_version: "kuduclusterkey@0"
I20260810 22:36:29.580737 31218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:36:29.602563 31218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:36:29.606858 31218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:36:29.607235 31218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:36:29.608096 31218 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:36:29.609282 31218 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:36:29.609391 31218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:36:29.609520 31218 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:36:29.609617 31218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:36:29.622426 31218 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.65:44989
I20260810 22:36:29.623832 31218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
I20260810 22:36:29.625236 31372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.65:44989 every 8 connection(s)
I20260810 22:36:29.625495 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 31218
I20260810 22:36:29.625648 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-0/wal/instance
I20260810 22:36:29.626020 17317 external_mini_cluster.cc:1428] Setting key ac28b8fdfedc3e20c718e3629d7bb78a
Aug 10 22:36:29 dist-test-slave-77v9 krb5kdc[30565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401389, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.65@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.233.66@KRBTEST.COM; defaulting to no policy
I20260810 22:36:29.651438 31376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:29.629054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:57079 (local address 127.16.233.126:40771)
0810 22:36:29.630829 (+ 1775us) server_negotiation.cc:206] Beginning negotiation
0810 22:36:29.630840 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 22:36:29.632157 (+ 1317us) server_negotiation.cc:407] Connection header received
0810 22:36:29.632695 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:36:29.632701 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:36:29.632781 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:36:29.632917 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:36:29.634625 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:29.635510 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:29.637371 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:29.637765 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:29.637855 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:29.641385 (+ 3530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:36:29.641390 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:36:29.641393 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:36:29.641426 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:36:29.643524 (+ 2098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:36:29.644286 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:36:29.644290 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:36:29.644292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:36:29.644393 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:36:29.649568 (+ 5175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:36:29.649575 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:36:29.649577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:36:29.649922 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:36:29.650036 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 22:36:29.650985 (+ 949us) server_negotiation.cc:299] Negotiation successful
0810 22:36:29.651175 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1669,"thread_start_us":102,"threads_started":1}
I20260810 22:36:29.652412 31377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:29.628526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:40771 (local address 127.16.233.65:57079)
0810 22:36:29.631125 (+ 2599us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:29.631193 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0810 22:36:29.632290 (+ 1097us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:36:29.633272 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:36:29.633281 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:36:29.633690 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:36:29.634434 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:29.634457 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:29.635695 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:29.635699 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:29.637209 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:29.637221 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:29.638106 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:29.638111 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:29.638343 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:29.639139 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:36:29.639162 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:36:29.641134 (+ 1972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:36:29.643706 (+ 2572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:36:29.643713 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:36:29.643733 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:36:29.644153 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:36:29.644504 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:36:29.644507 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:36:29.644509 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:36:29.644706 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:36:29.650151 (+ 5445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:36:29.650160 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:36:29.650575 (+ 415us) client_negotiation.cc:769] Sending connection context
0810 22:36:29.650828 (+ 253us) client_negotiation.cc:240] Negotiation successful
0810 22:36:29.651248 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2367,"thread_start_us":595,"threads_started":1}
I20260810 22:36:29.654012 31373 heartbeater.cc:347] Connected to a master server at 127.16.233.126:40771
I20260810 22:36:29.654436 31373 heartbeater.cc:464] Registering TS with master...
I20260810 22:36:29.655114 31373 heartbeater.cc:511] Master 127.16.233.126:40771 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.66@KRBTEST.COM" created.
I20260810 22:36:29.657159 31119 ts_manager.cc:194] Registered new tserver with Master: fc6f41614da44d429522b576d0ff0187 (127.16.233.65:44989)
I20260810 22:36:29.658851 31119 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.65@KRBTEST.COM'} at 127.16.233.65:57079
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab.
Entry for principal kudu/127.16.233.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab.
I20260810 22:36:29.680532 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.233.84:52919/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.66:0
--local_ip_for_outbound_sockets=127.16.233.66
--webserver_interface=127.16.233.66
--webserver_port=0
--tserver_master_addrs=127.16.233.126:40771
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab
--principal=kudu/127.16.233.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:35449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 22:36:29 dist-test-slave-77v9 krb5kdc[30565](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786401389, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260810 22:36:29.827128 31380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:36:29.827636 31380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:36:29.827776 31380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:36:29.832886 31380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:36:29.832979 31380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:36:29.833088 31380 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.66
I20260810 22:36:29.838831 31380 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:35449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.16.233.84:52919/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab
--principal=kudu/127.16.233.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.233.66
--webserver_port=0
--tserver_master_addrs=127.16.233.126:40771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31380
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.66
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:36:29.840169 31380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:36:29.841360 31380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:36:29.852253 31392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:36:29.852121 31390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:36:29.852134 31389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:36:29.853680 31380 server_base.cc:1034] running on GCE node
I20260810 22:36:29.854246 31380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:36:29.855018 31380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:36:29.856247 31380 hybrid_clock.cc:648] HybridClock initialized: now 1786401389856202 us; error 54 us; skew 500 ppm
Aug 10 22:36:29 dist-test-slave-77v9 krb5kdc[30565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401389, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:36:29.860102 31380 init.cc:377] Logged in from keytab as kudu/127.16.233.66@KRBTEST.COM (short username kudu)
I20260810 22:36:29.861445 31380 webserver.cc:460] Webserver started at http://127.16.233.66:33317/ using document root <none> and password file <none>
I20260810 22:36:29.862113 31380 fs_manager.cc:359] Metadata directory not provided
I20260810 22:36:29.862221 31380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:36:29.862478 31380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 22:36:29 dist-test-slave-77v9 krb5kdc[30565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401389, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.66@KRBTEST.COM for HTTP/127.16.233.84@KRBTEST.COM
I20260810 22:36:29.885290 31380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-1/data/instance:
uuid: "05f18aba3ea94033ad3641fce781f708"
format_stamp: "Formatted at 2026-08-10 22:36:29 on dist-test-slave-77v9"
server_key: "d45286c1c8f8fd718acae55122c27ffa"
server_key_iv: "7ddf68e462b062f7e9fbeff4be5654ba"
server_key_version: "kuduclusterkey@0"
I20260810 22:36:29.886106 31380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-1/wal/instance:
uuid: "05f18aba3ea94033ad3641fce781f708"
format_stamp: "Formatted at 2026-08-10 22:36:29 on dist-test-slave-77v9"
server_key: "d45286c1c8f8fd718acae55122c27ffa"
server_key_iv: "7ddf68e462b062f7e9fbeff4be5654ba"
server_key_version: "kuduclusterkey@0"
I20260810 22:36:29.891029 31380 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 22:36:29.908982 31399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:36:29.910921 31380 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.002s
I20260810 22:36:29.911100 31380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-1/wal
uuid: "05f18aba3ea94033ad3641fce781f708"
format_stamp: "Formatted at 2026-08-10 22:36:29 on dist-test-slave-77v9"
server_key: "d45286c1c8f8fd718acae55122c27ffa"
server_key_iv: "7ddf68e462b062f7e9fbeff4be5654ba"
server_key_version: "kuduclusterkey@0"
I20260810 22:36:29.911274 31380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:36:29.947424 31380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:36:29.951853 31380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:36:29.952082 31380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:36:29.953024 31380 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:36:29.954205 31380 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:36:29.954270 31380 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:36:29.954344 31380 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:36:29.954375 31380 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:36:29.965662 31380 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.66:38589
I20260810 22:36:29.965828 31528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.66:38589 every 8 connection(s)
I20260810 22:36:29.966912 31380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
I20260810 22:36:29.973543 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 31380
I20260810 22:36:29.973690 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-1/wal/instance
I20260810 22:36:29.974020 17317 external_mini_cluster.cc:1428] Setting key fe78acebe2d2d75ba0e0cf7b08e855d0
Aug 10 22:36:29 dist-test-slave-77v9 krb5kdc[30565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401389, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.66@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.233.67@KRBTEST.COM; defaulting to no policy
I20260810 22:36:29.989089 31376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:29.969286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:45347 (local address 127.16.233.126:40771)
0810 22:36:29.970924 (+ 1638us) server_negotiation.cc:206] Beginning negotiation
0810 22:36:29.970929 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:36:29.971464 (+ 535us) server_negotiation.cc:407] Connection header received
0810 22:36:29.972676 (+ 1212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:36:29.972680 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:36:29.972749 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:36:29.972931 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:36:29.977140 (+ 4209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:29.978007 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:29.979410 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:29.979839 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:29.979963 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:29.983576 (+ 3613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:36:29.983582 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:36:29.983589 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:36:29.983631 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:36:29.985759 (+ 2128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:36:29.986600 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:36:29.986606 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:36:29.986609 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:36:29.986711 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:36:29.987333 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:36:29.987340 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:36:29.987343 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:36:29.987662 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:36:29.987753 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 22:36:29.988656 (+ 903us) server_negotiation.cc:299] Negotiation successful
0810 22:36:29.988892 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1524}
I20260810 22:36:29.990074 31531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:29.969808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:40771 (local address 127.16.233.66:45347)
0810 22:36:29.971251 (+ 1443us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:29.971323 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0810 22:36:29.972403 (+ 1080us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:36:29.973123 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:36:29.973133 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:36:29.976102 (+ 2969us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:36:29.976926 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:29.976952 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:29.978205 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:29.978211 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:29.979234 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:29.979245 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:29.980192 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:29.980197 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:29.980447 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:29.981303 (+ 856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:36:29.981330 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:36:29.983296 (+ 1966us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:36:29.985959 (+ 2663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:36:29.985967 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:36:29.985992 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:36:29.986432 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:36:29.986858 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:36:29.986861 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:36:29.986866 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:36:29.987193 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:36:29.987869 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:36:29.987879 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:36:29.988299 (+ 420us) client_negotiation.cc:769] Sending connection context
0810 22:36:29.988554 (+ 255us) client_negotiation.cc:240] Negotiation successful
0810 22:36:29.988999 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1219,"thread_start_us":179,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.67@KRBTEST.COM" created.
I20260810 22:36:29.993402 31529 heartbeater.cc:347] Connected to a master server at 127.16.233.126:40771
I20260810 22:36:29.993724 31529 heartbeater.cc:464] Registering TS with master...
I20260810 22:36:29.994611 31529 heartbeater.cc:511] Master 127.16.233.126:40771 requested a full tablet report, sending...
I20260810 22:36:29.996048 31119 ts_manager.cc:194] Registered new tserver with Master: 05f18aba3ea94033ad3641fce781f708 (127.16.233.66:38589)
I20260810 22:36:29.996905 31119 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.66@KRBTEST.COM'} at 127.16.233.66:45347
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab.
Entry for principal kudu/127.16.233.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab.
I20260810 22:36:30.013401 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.233.84:52919/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.67:0
--local_ip_for_outbound_sockets=127.16.233.67
--webserver_interface=127.16.233.67
--webserver_port=0
--tserver_master_addrs=127.16.233.126:40771
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab
--principal=kudu/127.16.233.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:35449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:36:30.153909 31534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:36:30.154350 31534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:36:30.154452 31534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:36:30.159017 31534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:36:30.159139 31534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:36:30.159255 31534 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.67
I20260810 22:36:30.166802 31534 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:35449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.16.233.84:52919/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab
--principal=kudu/127.16.233.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.233.67
--webserver_port=0
--tserver_master_addrs=127.16.233.126:40771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31534
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.67
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:36:30.168061 31534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:36:30.169401 31534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:36:30.180253 31540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:36:30.180332 31543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:36:30.180706 31541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:36:30.180851 31534 server_base.cc:1034] running on GCE node
I20260810 22:36:30.181429 31534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:36:30.182160 31534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:36:30.189644 31534 hybrid_clock.cc:648] HybridClock initialized: now 1786401390189436 us; error 200 us; skew 500 ppm
Aug 10 22:36:30 dist-test-slave-77v9 krb5kdc[30565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401390, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:36:30.194265 31534 init.cc:377] Logged in from keytab as kudu/127.16.233.67@KRBTEST.COM (short username kudu)
I20260810 22:36:30.195739 31534 webserver.cc:460] Webserver started at http://127.16.233.67:33875/ using document root <none> and password file <none>
I20260810 22:36:30.196388 31534 fs_manager.cc:359] Metadata directory not provided
I20260810 22:36:30.196487 31534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:36:30.196722 31534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 22:36:30 dist-test-slave-77v9 krb5kdc[30565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401390, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.67@KRBTEST.COM for HTTP/127.16.233.84@KRBTEST.COM
I20260810 22:36:30.224300 31534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-2/data/instance:
uuid: "f66a65d1a2ec44c79932a15778993706"
format_stamp: "Formatted at 2026-08-10 22:36:30 on dist-test-slave-77v9"
server_key: "0a5c60399003b4c06ddf9362d742de1d"
server_key_iv: "c920b2214a95845eb29eddd2ef60b1c9"
server_key_version: "kuduclusterkey@0"
I20260810 22:36:30.225689 31534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-2/wal/instance:
uuid: "f66a65d1a2ec44c79932a15778993706"
format_stamp: "Formatted at 2026-08-10 22:36:30 on dist-test-slave-77v9"
server_key: "0a5c60399003b4c06ddf9362d742de1d"
server_key_iv: "c920b2214a95845eb29eddd2ef60b1c9"
server_key_version: "kuduclusterkey@0"
Aug 10 22:36:30 dist-test-slave-77v9 krb5kdc[30565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401389, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.126@KRBTEST.COM for HTTP/127.16.233.84@KRBTEST.COM
I20260810 22:36:30.237100 31534 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.006s sys 0.000s
I20260810 22:36:30.265954 31551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:36:30.267903 31534 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.002s sys 0.000s
I20260810 22:36:30.268060 31534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-2/wal
uuid: "f66a65d1a2ec44c79932a15778993706"
format_stamp: "Formatted at 2026-08-10 22:36:30 on dist-test-slave-77v9"
server_key: "0a5c60399003b4c06ddf9362d742de1d"
server_key_iv: "c920b2214a95845eb29eddd2ef60b1c9"
server_key_version: "kuduclusterkey@0"
I20260810 22:36:30.268203 31534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:36:30.284695 31534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:36:30.288661 31534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:36:30.288908 31534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:36:30.289664 31534 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:36:30.290836 31534 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:36:30.290952 31534 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:36:30.291055 31534 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:36:30.291122 31534 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:36:30.306828 31534 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.67:36923
I20260810 22:36:30.308220 31534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
I20260810 22:36:30.310318 31684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.67:36923 every 8 connection(s)
I20260810 22:36:30.313810 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 31534
I20260810 22:36:30.313949 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-2/wal/instance
I20260810 22:36:30.314263 17317 external_mini_cluster.cc:1428] Setting key 20764a13ba299eea47f5b948fd68f437
Aug 10 22:36:30 dist-test-slave-77v9 krb5kdc[30565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401390, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.67@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:36:30.339094 31376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:30.313032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:33025 (local address 127.16.233.126:40771)
0810 22:36:30.313174 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 22:36:30.313179 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:36:30.318854 (+ 5675us) server_negotiation.cc:407] Connection header received
0810 22:36:30.320187 (+ 1333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:36:30.320191 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:36:30.320262 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:36:30.320315 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:36:30.322132 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:30.322989 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:30.324983 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:30.329579 (+ 4596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:30.329667 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:30.334309 (+ 4642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:36:30.334313 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:36:30.334317 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:36:30.334349 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:36:30.336234 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:36:30.337079 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:36:30.337083 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:36:30.337085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:36:30.337190 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:36:30.337649 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:36:30.337653 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:36:30.337654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:36:30.337901 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:36:30.337986 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 22:36:30.338753 (+ 767us) server_negotiation.cc:299] Negotiation successful
0810 22:36:30.338915 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 22:36:30.343077 31688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:30.313282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:40771 (local address 127.16.233.67:33025)
0810 22:36:30.318673 (+ 5391us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:30.318742 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0810 22:36:30.319902 (+ 1160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:36:30.320773 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:36:30.320783 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:36:30.321219 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:36:30.321947 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:30.321967 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:30.323175 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:30.323179 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:30.324696 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:30.324748 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:30.330923 (+ 6175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:30.330927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:30.331219 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:30.332077 (+ 858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:36:30.332101 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:36:30.334044 (+ 1943us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:36:30.336469 (+ 2425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:36:30.336476 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:36:30.336498 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:36:30.336946 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:36:30.337320 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:36:30.337323 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:36:30.337325 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:36:30.337535 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:36:30.338056 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:36:30.338064 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:36:30.338447 (+ 383us) client_negotiation.cc:769] Sending connection context
0810 22:36:30.339639 (+ 1192us) client_negotiation.cc:240] Negotiation successful
0810 22:36:30.340057 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5129,"thread_start_us":3668,"threads_started":1}
I20260810 22:36:30.344748 31685 heartbeater.cc:347] Connected to a master server at 127.16.233.126:40771
I20260810 22:36:30.349239 31685 heartbeater.cc:464] Registering TS with master...
I20260810 22:36:30.349990 31685 heartbeater.cc:511] Master 127.16.233.126:40771 requested a full tablet report, sending...
I20260810 22:36:30.351374 31119 ts_manager.cc:194] Registered new tserver with Master: f66a65d1a2ec44c79932a15778993706 (127.16.233.67:36923)
I20260810 22:36:30.352152 31119 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.67@KRBTEST.COM'} at 127.16.233.67:33025
I20260810 22:36:30.354837 17317 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 22:36:30.355760 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 31218
I20260810 22:36:30.377214 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 31380
I20260810 22:36:30.387689 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 31534
I20260810 22:36:30.399996 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 31082
I20260810 22:36:30.410073 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:40771
--webserver_interface=127.16.233.126
--webserver_port=43393
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab
--principal=kudu/127.16.233.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:35449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.126:40771
--ranger_config_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.233.84:52919/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.233.84:52919/kms with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:36:30.531014 31690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:36:30.531406 31690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:36:30.531504 31690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:36:30.535880 31690 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:36:30.536032 31690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:36:30.536077 31690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:36:30.536119 31690 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:36:30.536182 31690 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:36:30.542165 31690 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:35449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.233.84:52919/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.16.233.126:40771
--ranger_config_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab
--principal=kudu/127.16.233.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:40771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.126
--webserver_port=43393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:36:30.543478 31690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:36:30.544682 31690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:36:30.554904 31696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:36:30.554890 31698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:36:30.554898 31695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:36:30.555330 31690 server_base.cc:1034] running on GCE node
I20260810 22:36:30.556010 31690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:36:30.557456 31690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:36:30.558686 31690 hybrid_clock.cc:648] HybridClock initialized: now 1786401390558657 us; error 52 us; skew 500 ppm
Aug 10 22:36:30 dist-test-slave-77v9 krb5kdc[30565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401390, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:36:30.562119 31690 init.cc:377] Logged in from keytab as kudu/127.16.233.126@KRBTEST.COM (short username kudu)
I20260810 22:36:30.563200 31690 webserver.cc:460] Webserver started at http://127.16.233.126:43393/ using document root <none> and password file <none>
I20260810 22:36:30.563796 31690 fs_manager.cc:359] Metadata directory not provided
I20260810 22:36:30.563906 31690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 22:36:30 dist-test-slave-77v9 krb5kdc[30565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401390, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.126@KRBTEST.COM for HTTP/127.16.233.84@KRBTEST.COM
I20260810 22:36:30.640775 31690 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 22:36:30.643529 31707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:36:30.645292 31690 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260810 22:36:30.645469 31690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/master-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/master-0/wal
uuid: "1176e1761df84e2b94862e38dd0a8d36"
format_stamp: "Formatted at 2026-08-10 22:36:28 on dist-test-slave-77v9"
server_key: "b69e222987b7c9eae15a2f0310247075"
server_key_iv: "10c1ecb1bdd3f0a9059c2c088d73bad8"
server_key_version: "kuduclusterkey@0"
I20260810 22:36:30.646087 31690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:36:30.670383 31690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:36:30.673738 31690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:36:30.674047 31690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:36:30.682301 31690 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.126:40771
I20260810 22:36:30.682314 31763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.126:40771 every 8 connection(s)
I20260810 22:36:30.683506 31690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
I20260810 22:36:30.688455 31764 sys_catalog.cc:263] Verifying existing consensus state
I20260810 22:36:30.690440 31764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1176e1761df84e2b94862e38dd0a8d36: Bootstrap starting.
I20260810 22:36:30.690692 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 31690
I20260810 22:36:30.691157 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.233.84:52919/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.65:44989
--local_ip_for_outbound_sockets=127.16.233.65
--tserver_master_addrs=127.16.233.126:40771
--webserver_port=41927
--webserver_interface=127.16.233.65
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab
--principal=kudu/127.16.233.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:35449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.233.84:52919/kms with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 22:36:30.709656 31764 log.cc:824] T 00000000000000000000000000000000 P 1176e1761df84e2b94862e38dd0a8d36: Log is configured to *not* fsync() on all Append() calls
I20260810 22:36:30.717413 31764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1176e1761df84e2b94862e38dd0a8d36: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 22:36:30.717923 31764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1176e1761df84e2b94862e38dd0a8d36: Bootstrap complete.
I20260810 22:36:30.722699 31764 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1176e1761df84e2b94862e38dd0a8d36 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1176e1761df84e2b94862e38dd0a8d36" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 40771 } }
I20260810 22:36:30.723213 31764 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1176e1761df84e2b94862e38dd0a8d36 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1176e1761df84e2b94862e38dd0a8d36, State: Initialized, Role: FOLLOWER
I20260810 22:36:30.723779 31764 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1176e1761df84e2b94862e38dd0a8d36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1176e1761df84e2b94862e38dd0a8d36" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 40771 } }
I20260810 22:36:30.723951 31764 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1176e1761df84e2b94862e38dd0a8d36 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:36:30.724053 31764 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1176e1761df84e2b94862e38dd0a8d36 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:36:30.724187 31764 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1176e1761df84e2b94862e38dd0a8d36 [term 1 FOLLOWER]: Advancing to term 2
I20260810 22:36:30.726576 31764 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1176e1761df84e2b94862e38dd0a8d36 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1176e1761df84e2b94862e38dd0a8d36" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 40771 } }
I20260810 22:36:30.726989 31764 leader_election.cc:302] T 00000000000000000000000000000000 P 1176e1761df84e2b94862e38dd0a8d36 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1176e1761df84e2b94862e38dd0a8d36; no voters:
I20260810 22:36:30.727388 31764 leader_election.cc:288] T 00000000000000000000000000000000 P 1176e1761df84e2b94862e38dd0a8d36 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 22:36:30.727548 31769 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1176e1761df84e2b94862e38dd0a8d36 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 22:36:30.727802 31769 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1176e1761df84e2b94862e38dd0a8d36 [term 2 LEADER]: Becoming Leader. State: Replica: 1176e1761df84e2b94862e38dd0a8d36, State: Running, Role: LEADER
I20260810 22:36:30.728205 31769 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1176e1761df84e2b94862e38dd0a8d36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1176e1761df84e2b94862e38dd0a8d36" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 40771 } }
I20260810 22:36:30.728894 31764 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1176e1761df84e2b94862e38dd0a8d36 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:36:30.729895 31770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1176e1761df84e2b94862e38dd0a8d36 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1176e1761df84e2b94862e38dd0a8d36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1176e1761df84e2b94862e38dd0a8d36" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 40771 } } }
I20260810 22:36:30.730054 31770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1176e1761df84e2b94862e38dd0a8d36 [sys.catalog]: This master's current role is: LEADER
I20260810 22:36:30.730496 31771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1176e1761df84e2b94862e38dd0a8d36 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1176e1761df84e2b94862e38dd0a8d36. Latest consensus state: current_term: 2 leader_uuid: "1176e1761df84e2b94862e38dd0a8d36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1176e1761df84e2b94862e38dd0a8d36" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 40771 } } }
I20260810 22:36:30.730652 31771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1176e1761df84e2b94862e38dd0a8d36 [sys.catalog]: This master's current role is: LEADER
I20260810 22:36:30.730937 31777 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:36:30.734364 31777 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:36:30.734992 31764 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 22:36:30.735193 31777 catalog_manager.cc:1259] Loaded cluster ID: f13b3238a3b34e0a84d7b302b6d6f075
I20260810 22:36:30.735312 31777 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:36:30.741528 31777 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:36:30.743772 31777 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1176e1761df84e2b94862e38dd0a8d36: Loaded TSK: 0
I20260810 22:36:30.745615 31777 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 22:36:30.857257 31767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:36:30.857590 31767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:36:30.857690 31767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:36:30.863312 31767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:36:30.863529 31767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:36:30.863675 31767 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.65
I20260810 22:36:30.871214 31767 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:35449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.233.84:52919/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab
--principal=kudu/127.16.233.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.65:44989
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.233.65
--webserver_port=41927
--tserver_master_addrs=127.16.233.126:40771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.65
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:36:30.872601 31767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:36:30.873787 31767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:36:30.884967 31807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:36:30.885020 31809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:36:30.885459 31767 server_base.cc:1034] running on GCE node
W20260810 22:36:30.885003 31806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:36:30.886083 31767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:36:30.886907 31767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:36:30.888208 31767 hybrid_clock.cc:648] HybridClock initialized: now 1786401390888167 us; error 52 us; skew 500 ppm
Aug 10 22:36:30 dist-test-slave-77v9 krb5kdc[30565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401390, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:36:30.891534 31767 init.cc:377] Logged in from keytab as kudu/127.16.233.65@KRBTEST.COM (short username kudu)
I20260810 22:36:30.892844 31767 webserver.cc:460] Webserver started at http://127.16.233.65:41927/ using document root <none> and password file <none>
I20260810 22:36:30.893688 31767 fs_manager.cc:359] Metadata directory not provided
I20260810 22:36:30.893824 31767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 22:36:30 dist-test-slave-77v9 krb5kdc[30565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401390, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.65@KRBTEST.COM for HTTP/127.16.233.84@KRBTEST.COM
I20260810 22:36:30.925796 31767 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 22:36:30.929855 31817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:36:30.931813 31767 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260810 22:36:30.931986 31767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-0/wal
uuid: "fc6f41614da44d429522b576d0ff0187"
format_stamp: "Formatted at 2026-08-10 22:36:29 on dist-test-slave-77v9"
server_key: "860292d7d4f6140aed32c948b7519da0"
server_key_iv: "4644537329984c7c51a7d21b29e68386"
server_key_version: "kuduclusterkey@0"
I20260810 22:36:30.932878 31767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:36:30.948945 31767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:36:30.953972 31767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:36:30.954329 31767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:36:30.955300 31767 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:36:30.956668 31767 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:36:30.956746 31767 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:36:30.956848 31767 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:36:30.956929 31767 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260810 22:36:30.970724 31767 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.65:44989
I20260810 22:36:30.971074 31945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.65:44989 every 8 connection(s)
I20260810 22:36:30.972302 31767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
I20260810 22:36:30.981400 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 31767
I20260810 22:36:30.981974 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.233.84:52919/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.66:38589
--local_ip_for_outbound_sockets=127.16.233.66
--tserver_master_addrs=127.16.233.126:40771
--webserver_port=33317
--webserver_interface=127.16.233.66
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab
--principal=kudu/127.16.233.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:35449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.233.84:52919/kms with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 22:36:30 dist-test-slave-77v9 krb5kdc[30565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401390, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.65@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:36:31.002362 31949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:30.975198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:33805 (local address 127.16.233.126:40771)
0810 22:36:30.975869 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0810 22:36:30.975904 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0810 22:36:30.976409 (+ 505us) server_negotiation.cc:407] Connection header received
0810 22:36:30.978063 (+ 1654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:36:30.978074 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:36:30.978564 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:36:30.978957 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:36:30.980384 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:30.985401 (+ 5017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:30.987478 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:30.988251 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:30.988448 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:30.993658 (+ 5210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:36:30.993666 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:36:30.993688 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:36:30.993730 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:36:30.996258 (+ 2528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:36:30.997149 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:36:30.997156 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:36:30.997162 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:36:30.997317 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:36:30.997833 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:36:30.997837 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:36:30.997839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:36:30.998439 (+ 600us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:36:30.998584 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 22:36:31.001069 (+ 2485us) server_negotiation.cc:299] Negotiation successful
0810 22:36:31.001408 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":133,"threads_started":1}
I20260810 22:36:31.000782 31950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:30.975299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:40771 (local address 127.16.233.65:33805)
0810 22:36:30.976197 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:30.976260 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0810 22:36:30.977476 (+ 1216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:36:30.979088 (+ 1612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:36:30.979097 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:36:30.979488 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:36:30.980132 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:30.980153 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:30.985663 (+ 5510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:30.985667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:30.987273 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:30.987292 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:30.988735 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:30.988740 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:30.988983 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:30.991143 (+ 2160us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:36:30.991168 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:36:30.993311 (+ 2143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:36:30.996446 (+ 3135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:36:30.996454 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:36:30.996473 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:36:30.996980 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:36:30.997443 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:36:30.997446 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:36:30.997448 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:36:30.997704 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:36:30.998650 (+ 946us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:36:30.998657 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:36:30.999001 (+ 344us) client_negotiation.cc:769] Sending connection context
0810 22:36:30.999214 (+ 213us) client_negotiation.cc:240] Negotiation successful
0810 22:36:30.999603 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"thread_start_us":156,"threads_started":1}
I20260810 22:36:31.006655 31947 heartbeater.cc:347] Connected to a master server at 127.16.233.126:40771
I20260810 22:36:31.006946 31947 heartbeater.cc:464] Registering TS with master...
I20260810 22:36:31.007586 31947 heartbeater.cc:511] Master 127.16.233.126:40771 requested a full tablet report, sending...
W20260810 22:36:31.010295 31947 heartbeater.cc:644] Failed to heartbeat to 127.16.233.126:40771 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 22:36:31.010457 31947 heartbeater.cc:464] Registering TS with master...
I20260810 22:36:31.010663 31947 heartbeater.cc:511] Master 127.16.233.126:40771 requested a full tablet report, sending...
I20260810 22:36:31.011603 31947 heartbeater.cc:464] Registering TS with master...
I20260810 22:36:31.011816 31947 heartbeater.cc:511] Master 127.16.233.126:40771 requested a full tablet report, sending...
W20260810 22:36:31.012667 31947 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 22:36:31.138811 31951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:36:31.139200 31951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:36:31.139290 31951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:36:31.144120 31951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:36:31.144207 31951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:36:31.144311 31951 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.66
I20260810 22:36:31.152275 31951 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:35449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.233.84:52919/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab
--principal=kudu/127.16.233.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.66:38589
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.233.66
--webserver_port=33317
--tserver_master_addrs=127.16.233.126:40771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.66
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:36:31.153863 31951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:36:31.155316 31951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:36:31.166811 31956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:36:31.166838 31957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:36:31.166867 31960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:36:31.167279 31951 server_base.cc:1034] running on GCE node
I20260810 22:36:31.167871 31951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:36:31.168711 31951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:36:31.169936 31951 hybrid_clock.cc:648] HybridClock initialized: now 1786401391169875 us; error 60 us; skew 500 ppm
Aug 10 22:36:31 dist-test-slave-77v9 krb5kdc[30565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401391, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:36:31.173121 31951 init.cc:377] Logged in from keytab as kudu/127.16.233.66@KRBTEST.COM (short username kudu)
I20260810 22:36:31.174228 31951 webserver.cc:460] Webserver started at http://127.16.233.66:33317/ using document root <none> and password file <none>
I20260810 22:36:31.174827 31951 fs_manager.cc:359] Metadata directory not provided
I20260810 22:36:31.174949 31951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 22:36:31 dist-test-slave-77v9 krb5kdc[30565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401391, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.66@KRBTEST.COM for HTTP/127.16.233.84@KRBTEST.COM
I20260810 22:36:31.202363 31951 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.003s sys 0.001s
I20260810 22:36:31.207953 31970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:36:31.210112 31951 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.003s sys 0.000s
I20260810 22:36:31.210378 31951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-1/wal
uuid: "05f18aba3ea94033ad3641fce781f708"
format_stamp: "Formatted at 2026-08-10 22:36:29 on dist-test-slave-77v9"
server_key: "d45286c1c8f8fd718acae55122c27ffa"
server_key_iv: "7ddf68e462b062f7e9fbeff4be5654ba"
server_key_version: "kuduclusterkey@0"
I20260810 22:36:31.211445 31951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:36:31.243641 31951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:36:31.247974 31951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:36:31.248236 31951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:36:31.249172 31951 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:36:31.250489 31951 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:36:31.250577 31951 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:36:31.250653 31951 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:36:31.250689 31951 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:36:31.263876 31951 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.66:38589
I20260810 22:36:31.263935 32098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.66:38589 every 8 connection(s)
I20260810 22:36:31.265523 31951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
I20260810 22:36:31.268599 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 31951
I20260810 22:36:31.269218 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.233.84:52919/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.67:36923
--local_ip_for_outbound_sockets=127.16.233.67
--tserver_master_addrs=127.16.233.126:40771
--webserver_port=33875
--webserver_interface=127.16.233.67
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab
--principal=kudu/127.16.233.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:35449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.233.84:52919/kms with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 22:36:31 dist-test-slave-77v9 krb5kdc[30565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401391, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.66@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:36:31.294296 31949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:31.269050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:52959 (local address 127.16.233.126:40771)
0810 22:36:31.269198 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 22:36:31.269204 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:36:31.276314 (+ 7110us) server_negotiation.cc:407] Connection header received
0810 22:36:31.277878 (+ 1564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:36:31.277883 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:36:31.277962 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:36:31.278022 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:36:31.280100 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:31.281019 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:31.283258 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:31.283626 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:31.283723 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:31.287328 (+ 3605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:36:31.287331 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:36:31.287334 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:36:31.287367 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:36:31.289286 (+ 1919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:36:31.291526 (+ 2240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:36:31.291533 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:36:31.291538 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:36:31.291664 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:36:31.292224 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:36:31.292228 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:36:31.292230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:36:31.292499 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:36:31.292609 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 22:36:31.293876 (+ 1267us) server_negotiation.cc:299] Negotiation successful
0810 22:36:31.294077 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 22:36:31.300411 32101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:31.269380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:40771 (local address 127.16.233.66:52959)
0810 22:36:31.276058 (+ 6678us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:31.276160 (+ 102us) client_negotiation.cc:174] Beginning negotiation
0810 22:36:31.277534 (+ 1374us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:36:31.278530 (+ 996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:36:31.278541 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:36:31.278981 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:36:31.279868 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:31.279893 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:31.281476 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:31.281479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:31.283071 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:31.283085 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:31.283999 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:31.284003 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:31.284232 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:31.285138 (+ 906us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:36:31.285160 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:36:31.287073 (+ 1913us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:36:31.290778 (+ 3705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:36:31.290788 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:36:31.290812 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:36:31.291347 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:36:31.291805 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:36:31.291809 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:36:31.291811 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:36:31.292081 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:36:31.292674 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:36:31.292682 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:36:31.293117 (+ 435us) client_negotiation.cc:769] Sending connection context
0810 22:36:31.293373 (+ 256us) client_negotiation.cc:240] Negotiation successful
0810 22:36:31.295361 (+ 1988us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6325,"thread_start_us":158,"threads_started":1}
I20260810 22:36:31.302659 32099 heartbeater.cc:347] Connected to a master server at 127.16.233.126:40771
I20260810 22:36:31.303089 32099 heartbeater.cc:464] Registering TS with master...
I20260810 22:36:31.303815 32099 heartbeater.cc:511] Master 127.16.233.126:40771 requested a full tablet report, sending...
W20260810 22:36:31.306051 32099 heartbeater.cc:644] Failed to heartbeat to 127.16.233.126:40771 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 22:36:31.306208 32099 heartbeater.cc:464] Registering TS with master...
I20260810 22:36:31.306489 32099 heartbeater.cc:511] Master 127.16.233.126:40771 requested a full tablet report, sending...
I20260810 22:36:31.307812 32099 heartbeater.cc:464] Registering TS with master...
I20260810 22:36:31.308073 32099 heartbeater.cc:511] Master 127.16.233.126:40771 requested a full tablet report, sending...
W20260810 22:36:31.309183 32099 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 22:36:31.437724 32102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:36:31.438025 32102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:36:31.438071 32102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:36:31.442739 32102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:36:31.442849 32102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:36:31.442955 32102 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.67
I20260810 22:36:31.450171 32102 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:35449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.233.84:52919/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab
--principal=kudu/127.16.233.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.67:36923
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.233.67
--webserver_port=33875
--tserver_master_addrs=127.16.233.126:40771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.67
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:36:31.451918 32102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:36:31.453505 32102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:36:31.467770 32109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:36:31.468561 32114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:36:31.468989 32110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:36:31.470340 32102 server_base.cc:1034] running on GCE node
I20260810 22:36:31.470947 32102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:36:31.471839 32102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:36:31.473052 32102 hybrid_clock.cc:648] HybridClock initialized: now 1786401391473025 us; error 51 us; skew 500 ppm
Aug 10 22:36:31 dist-test-slave-77v9 krb5kdc[30565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401391, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:36:31.477272 32102 init.cc:377] Logged in from keytab as kudu/127.16.233.67@KRBTEST.COM (short username kudu)
I20260810 22:36:31.478616 32102 webserver.cc:460] Webserver started at http://127.16.233.67:33875/ using document root <none> and password file <none>
I20260810 22:36:31.479380 32102 fs_manager.cc:359] Metadata directory not provided
I20260810 22:36:31.479506 32102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 22:36:31 dist-test-slave-77v9 krb5kdc[30565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401391, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.67@KRBTEST.COM for HTTP/127.16.233.84@KRBTEST.COM
I20260810 22:36:31.505759 32102 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 22:36:31.509307 32122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:36:31.510870 32102 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 22:36:31.511031 32102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-2/wal
uuid: "f66a65d1a2ec44c79932a15778993706"
format_stamp: "Formatted at 2026-08-10 22:36:30 on dist-test-slave-77v9"
server_key: "0a5c60399003b4c06ddf9362d742de1d"
server_key_iv: "c920b2214a95845eb29eddd2ef60b1c9"
server_key_version: "kuduclusterkey@0"
I20260810 22:36:31.511646 32102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:36:31.530838 32102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:36:31.534745 32102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:36:31.535002 32102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:36:31.535852 32102 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:36:31.536978 32102 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:36:31.537060 32102 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:36:31.537155 32102 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:36:31.537214 32102 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:36:31.549723 32102 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.67:36923
I20260810 22:36:31.551049 32102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
I20260810 22:36:31.552420 32252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.67:36923 every 8 connection(s)
I20260810 22:36:31.555840 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 32102
Aug 10 22:36:31 dist-test-slave-77v9 krb5kdc[30565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401391, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.67@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:36:31.568390 31949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:31.553292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:56017 (local address 127.16.233.126:40771)
0810 22:36:31.553433 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 22:36:31.553437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:36:31.554433 (+ 996us) server_negotiation.cc:407] Connection header received
0810 22:36:31.555602 (+ 1169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:36:31.555605 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:36:31.555671 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:36:31.555750 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:36:31.557498 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:31.558259 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:31.560042 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:31.560369 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:31.560478 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:31.563924 (+ 3446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:36:31.563927 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:36:31.563930 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:36:31.564018 (+ 88us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:36:31.565825 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:36:31.566497 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:36:31.566502 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:36:31.566503 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:36:31.566609 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:36:31.567014 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:36:31.567019 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:36:31.567020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:36:31.567280 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:36:31.567403 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 22:36:31.568081 (+ 678us) server_negotiation.cc:299] Negotiation successful
0810 22:36:31.568220 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 22:36:31.569501 32255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:31.553774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:40771 (local address 127.16.233.67:56017)
0810 22:36:31.554264 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:31.554320 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 22:36:31.555345 (+ 1025us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:36:31.555937 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:36:31.555956 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:36:31.556268 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:36:31.557271 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:31.557285 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:31.558406 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:31.558411 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:31.559898 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:31.559909 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:31.560664 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:31.560668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:31.560904 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:31.561688 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:36:31.561715 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:36:31.563761 (+ 2046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:36:31.565934 (+ 2173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:36:31.565943 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:36:31.565964 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:36:31.566378 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:36:31.566725 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:36:31.566728 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:36:31.566730 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:36:31.566898 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:36:31.567404 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:36:31.567412 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:36:31.567783 (+ 371us) client_negotiation.cc:769] Sending connection context
0810 22:36:31.568042 (+ 259us) client_negotiation.cc:240] Negotiation successful
0810 22:36:31.568400 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":145,"threads_started":1}
I20260810 22:36:31.571061 32253 heartbeater.cc:347] Connected to a master server at 127.16.233.126:40771
I20260810 22:36:31.571381 32253 heartbeater.cc:464] Registering TS with master...
I20260810 22:36:31.571609 32256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:31.556629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49932 (local address 127.16.233.126:40771)
0810 22:36:31.557622 (+ 993us) server_negotiation.cc:206] Beginning negotiation
0810 22:36:31.557626 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:36:31.557895 (+ 269us) server_negotiation.cc:407] Connection header received
0810 22:36:31.558094 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:36:31.558097 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:36:31.558144 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:36:31.558199 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:36:31.559276 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:31.560015 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:31.561249 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:31.561552 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:31.561631 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:31.563457 (+ 1826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:36:31.563461 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:36:31.563464 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:36:31.563491 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:36:31.565624 (+ 2133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:36:31.566508 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:36:31.566512 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:36:31.566513 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:36:31.566619 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:36:31.567446 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:36:31.567449 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:36:31.567451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:36:31.567634 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:36:31.567829 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0810 22:36:31.571223 (+ 3394us) server_negotiation.cc:299] Negotiation successful
0810 22:36:31.571375 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":919,"spinlock_wait_cycles":768,"thread_start_us":99,"threads_started":1}
I20260810 22:36:31.572094 32253 heartbeater.cc:511] Master 127.16.233.126:40771 requested a full tablet report, sending...
W20260810 22:36:31.574280 32253 heartbeater.cc:644] Failed to heartbeat to 127.16.233.126:40771 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 22:36:31.574446 32253 heartbeater.cc:464] Registering TS with master...
I20260810 22:36:31.574702 32253 heartbeater.cc:511] Master 127.16.233.126:40771 requested a full tablet report, sending...
I20260810 22:36:31.575594 32253 heartbeater.cc:464] Registering TS with master...
I20260810 22:36:31.575800 32253 heartbeater.cc:511] Master 127.16.233.126:40771 requested a full tablet report, sending...
W20260810 22:36:31.576601 32253 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 10 22:36:31 dist-test-slave-77v9 krb5kdc[30565](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786401391, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:36:32.013001 31947 heartbeater.cc:464] Registering TS with master...
I20260810 22:36:32.013375 31947 heartbeater.cc:511] Master 127.16.233.126:40771 requested a full tablet report, sending...
I20260810 22:36:32.014849 31725 ts_manager.cc:194] Registered new tserver with Master: fc6f41614da44d429522b576d0ff0187 (127.16.233.65:44989)
I20260810 22:36:32.016678 31725 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.65@KRBTEST.COM'} at 127.16.233.65:33805
I20260810 22:36:32.309461 32099 heartbeater.cc:464] Registering TS with master...
I20260810 22:36:32.309880 32099 heartbeater.cc:511] Master 127.16.233.126:40771 requested a full tablet report, sending...
I20260810 22:36:32.310855 31724 ts_manager.cc:194] Registered new tserver with Master: 05f18aba3ea94033ad3641fce781f708 (127.16.233.66:38589)
I20260810 22:36:32.311691 31724 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.66@KRBTEST.COM'} at 127.16.233.66:52959
Aug 10 22:36:32 dist-test-slave-77v9 krb5kdc[30565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401391, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.126@KRBTEST.COM for HTTP/127.16.233.84@KRBTEST.COM
I20260810 22:36:32.576930 32253 heartbeater.cc:464] Registering TS with master...
I20260810 22:36:32.577311 32253 heartbeater.cc:511] Master 127.16.233.126:40771 requested a full tablet report, sending...
I20260810 22:36:32.578414 31724 ts_manager.cc:194] Registered new tserver with Master: f66a65d1a2ec44c79932a15778993706 (127.16.233.67:36923)
I20260810 22:36:32.579301 31724 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.67@KRBTEST.COM'} at 127.16.233.67:56017
I20260810 22:36:32.584250 17317 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 22:36:32.597566 32277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:32.586417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49940 (local address 127.16.233.126:40771)
0810 22:36:32.586731 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0810 22:36:32.586737 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:36:32.586783 (+ 46us) server_negotiation.cc:407] Connection header received
0810 22:36:32.586953 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:36:32.586956 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:36:32.587015 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:36:32.587134 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:36:32.588085 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:32.588979 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:32.590273 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:32.590657 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:32.590783 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:32.592859 (+ 2076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:36:32.592864 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:36:32.592868 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:36:32.592909 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:36:32.595185 (+ 2276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:36:32.595833 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:36:32.595837 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:36:32.595839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:36:32.595961 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:36:32.596383 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:36:32.596388 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:36:32.596390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:36:32.596666 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:36:32.596867 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0810 22:36:32.597162 (+ 295us) server_negotiation.cc:299] Negotiation successful
0810 22:36:32.597344 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":116,"threads_started":1}
I20260810 22:36:32.602838 31724 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49940:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 22:36:32.608144 31724 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:36:32.631901 32286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:32.623287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.67:36923 (local address 127.0.0.1:52448)
0810 22:36:32.625663 (+ 2376us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:32.625740 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0810 22:36:32.625906 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:36:32.626785 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:36:32.626793 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:36:32.626817 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:36:32.626966 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:32.626973 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:32.628466 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:32.628469 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:32.629948 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:32.629961 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:32.631314 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:32.631317 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:32.631444 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:32.631490 (+ 46us) client_negotiation.cc:769] Sending connection context
0810 22:36:32.631604 (+ 114us) client_negotiation.cc:240] Negotiation successful
0810 22:36:32.631733 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1942,"thread_start_us":161,"threads_started":1}
I20260810 22:36:32.633188 32285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:32.623444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52448 (local address 127.16.233.67:36923)
0810 22:36:32.623863 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0810 22:36:32.623872 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 22:36:32.626075 (+ 2203us) server_negotiation.cc:407] Connection header received
0810 22:36:32.626168 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:36:32.626174 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:36:32.626432 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:36:32.626598 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:36:32.627116 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:32.628321 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:32.630102 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:32.630978 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:32.631085 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:32.631241 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0810 22:36:32.632922 (+ 1681us) server_negotiation.cc:299] Negotiation successful
0810 22:36:32.633040 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":144,"threads_started":1}
I20260810 22:36:32.635738 32284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:32.624680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.65:44989 (local address 127.0.0.1:44108)
0810 22:36:32.627221 (+ 2541us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:32.627236 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:36:32.627359 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:36:32.628165 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:36:32.628168 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:36:32.628187 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:36:32.628352 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:32.628360 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:32.631048 (+ 2688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:32.631051 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:32.632520 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:32.632533 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:32.635215 (+ 2682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:32.635219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:32.635332 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:32.635394 (+ 62us) client_negotiation.cc:769] Sending connection context
0810 22:36:32.635473 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 22:36:32.635550 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2475,"thread_start_us":356,"threads_started":1}
I20260810 22:36:32.636330 32289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:32.625764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44108 (local address 127.16.233.65:44989)
0810 22:36:32.627035 (+ 1271us) server_negotiation.cc:206] Beginning negotiation
0810 22:36:32.627044 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 22:36:32.627466 (+ 422us) server_negotiation.cc:407] Connection header received
0810 22:36:32.627558 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:36:32.627567 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:36:32.627808 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:36:32.627973 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:36:32.628511 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:32.630890 (+ 2379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:32.632723 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:32.634893 (+ 2170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:32.635003 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:32.635149 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 22:36:32.636062 (+ 913us) server_negotiation.cc:299] Negotiation successful
0810 22:36:32.636177 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1149,"thread_start_us":174,"threads_started":1}
I20260810 22:36:32.637270 32154 tablet_service.cc:1467] Processing CreateTablet for tablet 7c6fe54437264fc88d05b8746d9f327d (DEFAULT_TABLE table=test-table [id=75d4d210da684135b4d1bd5b9f7d13e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:36:32.638685 32154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c6fe54437264fc88d05b8746d9f327d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:36:32.639871 32286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:32.625192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.66:38589 (local address 127.0.0.1:48972)
0810 22:36:32.632674 (+ 7482us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:32.632690 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 22:36:32.632764 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:36:32.634548 (+ 1784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:36:32.634551 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:36:32.634571 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:36:32.634657 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:32.634662 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:32.636115 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:32.636118 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:32.637286 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:32.637298 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:32.639423 (+ 2125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:32.639427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:32.639530 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:32.639559 (+ 29us) client_negotiation.cc:769] Sending connection context
0810 22:36:32.639627 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 22:36:32.639721 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7416,"thread_start_us":231,"threads_started":1}
I20260810 22:36:32.640671 32288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:32.625583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48972 (local address 127.16.233.66:38589)
0810 22:36:32.633978 (+ 8395us) server_negotiation.cc:206] Beginning negotiation
0810 22:36:32.633986 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 22:36:32.634014 (+ 28us) server_negotiation.cc:407] Connection header received
0810 22:36:32.634097 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:36:32.634102 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:36:32.634335 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:36:32.634514 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:36:32.634792 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:32.635977 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:32.637446 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:32.638282 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:32.640160 (+ 1878us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:32.640369 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0810 22:36:32.640428 (+ 59us) server_negotiation.cc:299] Negotiation successful
0810 22:36:32.640532 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8234,"thread_start_us":92,"threads_started":1}
I20260810 22:36:32.640264 31868 tablet_service.cc:1467] Processing CreateTablet for tablet 7c6fe54437264fc88d05b8746d9f327d (DEFAULT_TABLE table=test-table [id=75d4d210da684135b4d1bd5b9f7d13e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:36:32.642433 31868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c6fe54437264fc88d05b8746d9f327d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:36:32.643831 32022 tablet_service.cc:1467] Processing CreateTablet for tablet 7c6fe54437264fc88d05b8746d9f327d (DEFAULT_TABLE table=test-table [id=75d4d210da684135b4d1bd5b9f7d13e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:36:32.645088 32022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c6fe54437264fc88d05b8746d9f327d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:36:32.652076 32291 tablet_bootstrap.cc:492] T 7c6fe54437264fc88d05b8746d9f327d P fc6f41614da44d429522b576d0ff0187: Bootstrap starting.
I20260810 22:36:32.657618 32291 tablet_bootstrap.cc:654] T 7c6fe54437264fc88d05b8746d9f327d P fc6f41614da44d429522b576d0ff0187: Neither blocks nor log segments found. Creating new log.
I20260810 22:36:32.659118 32291 log.cc:824] T 7c6fe54437264fc88d05b8746d9f327d P fc6f41614da44d429522b576d0ff0187: Log is configured to *not* fsync() on all Append() calls
I20260810 22:36:32.659461 32290 tablet_bootstrap.cc:492] T 7c6fe54437264fc88d05b8746d9f327d P f66a65d1a2ec44c79932a15778993706: Bootstrap starting.
I20260810 22:36:32.660105 32292 tablet_bootstrap.cc:492] T 7c6fe54437264fc88d05b8746d9f327d P 05f18aba3ea94033ad3641fce781f708: Bootstrap starting.
I20260810 22:36:32.663400 32290 tablet_bootstrap.cc:654] T 7c6fe54437264fc88d05b8746d9f327d P f66a65d1a2ec44c79932a15778993706: Neither blocks nor log segments found. Creating new log.
I20260810 22:36:32.663400 32292 tablet_bootstrap.cc:654] T 7c6fe54437264fc88d05b8746d9f327d P 05f18aba3ea94033ad3641fce781f708: Neither blocks nor log segments found. Creating new log.
I20260810 22:36:32.664906 32292 log.cc:824] T 7c6fe54437264fc88d05b8746d9f327d P 05f18aba3ea94033ad3641fce781f708: Log is configured to *not* fsync() on all Append() calls
I20260810 22:36:32.664906 32290 log.cc:824] T 7c6fe54437264fc88d05b8746d9f327d P f66a65d1a2ec44c79932a15778993706: Log is configured to *not* fsync() on all Append() calls
I20260810 22:36:32.668543 32292 tablet_bootstrap.cc:492] T 7c6fe54437264fc88d05b8746d9f327d P 05f18aba3ea94033ad3641fce781f708: No bootstrap required, opened a new log
I20260810 22:36:32.669761 32290 tablet_bootstrap.cc:492] T 7c6fe54437264fc88d05b8746d9f327d P f66a65d1a2ec44c79932a15778993706: No bootstrap required, opened a new log
I20260810 22:36:32.670030 32290 ts_tablet_manager.cc:1397] T 7c6fe54437264fc88d05b8746d9f327d P f66a65d1a2ec44c79932a15778993706: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.003s
I20260810 22:36:32.670125 32291 tablet_bootstrap.cc:492] T 7c6fe54437264fc88d05b8746d9f327d P fc6f41614da44d429522b576d0ff0187: No bootstrap required, opened a new log
I20260810 22:36:32.670336 32291 ts_tablet_manager.cc:1397] T 7c6fe54437264fc88d05b8746d9f327d P fc6f41614da44d429522b576d0ff0187: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.001s
I20260810 22:36:32.677219 32291 raft_consensus.cc:348] T 7c6fe54437264fc88d05b8746d9f327d P fc6f41614da44d429522b576d0ff0187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66a65d1a2ec44c79932a15778993706" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 36923 } } peers { permanent_uuid: "fc6f41614da44d429522b576d0ff0187" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 44989 } } peers { permanent_uuid: "05f18aba3ea94033ad3641fce781f708" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 38589 } }
I20260810 22:36:32.677812 32291 raft_consensus.cc:374] T 7c6fe54437264fc88d05b8746d9f327d P fc6f41614da44d429522b576d0ff0187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:36:32.677940 32291 raft_consensus.cc:729] T 7c6fe54437264fc88d05b8746d9f327d P fc6f41614da44d429522b576d0ff0187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc6f41614da44d429522b576d0ff0187, State: Initialized, Role: FOLLOWER
I20260810 22:36:32.678781 32291 consensus_queue.cc:260] T 7c6fe54437264fc88d05b8746d9f327d P fc6f41614da44d429522b576d0ff0187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66a65d1a2ec44c79932a15778993706" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 36923 } } peers { permanent_uuid: "fc6f41614da44d429522b576d0ff0187" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 44989 } } peers { permanent_uuid: "05f18aba3ea94033ad3641fce781f708" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 38589 } }
I20260810 22:36:32.678561 32290 raft_consensus.cc:348] T 7c6fe54437264fc88d05b8746d9f327d P f66a65d1a2ec44c79932a15778993706 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66a65d1a2ec44c79932a15778993706" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 36923 } } peers { permanent_uuid: "fc6f41614da44d429522b576d0ff0187" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 44989 } } peers { permanent_uuid: "05f18aba3ea94033ad3641fce781f708" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 38589 } }
I20260810 22:36:32.679152 32290 raft_consensus.cc:374] T 7c6fe54437264fc88d05b8746d9f327d P f66a65d1a2ec44c79932a15778993706 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:36:32.679256 32290 raft_consensus.cc:729] T 7c6fe54437264fc88d05b8746d9f327d P f66a65d1a2ec44c79932a15778993706 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f66a65d1a2ec44c79932a15778993706, State: Initialized, Role: FOLLOWER
I20260810 22:36:32.679893 32290 consensus_queue.cc:260] T 7c6fe54437264fc88d05b8746d9f327d P f66a65d1a2ec44c79932a15778993706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66a65d1a2ec44c79932a15778993706" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 36923 } } peers { permanent_uuid: "fc6f41614da44d429522b576d0ff0187" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 44989 } } peers { permanent_uuid: "05f18aba3ea94033ad3641fce781f708" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 38589 } }
I20260810 22:36:32.681326 32290 ts_tablet_manager.cc:1428] T 7c6fe54437264fc88d05b8746d9f327d P f66a65d1a2ec44c79932a15778993706: Time spent starting tablet: real 0.011s user 0.007s sys 0.004s
I20260810 22:36:32.681988 32253 heartbeater.cc:503] Master 127.16.233.126:40771 was elected leader, sending a full tablet report...
I20260810 22:36:32.697106 32291 ts_tablet_manager.cc:1428] T 7c6fe54437264fc88d05b8746d9f327d P fc6f41614da44d429522b576d0ff0187: Time spent starting tablet: real 0.027s user 0.007s sys 0.003s
I20260810 22:36:32.697679 31947 heartbeater.cc:503] Master 127.16.233.126:40771 was elected leader, sending a full tablet report...
I20260810 22:36:32.701408 32292 ts_tablet_manager.cc:1397] T 7c6fe54437264fc88d05b8746d9f327d P 05f18aba3ea94033ad3641fce781f708: Time spent bootstrapping tablet: real 0.041s user 0.008s sys 0.000s
I20260810 22:36:32.718947 32292 raft_consensus.cc:348] T 7c6fe54437264fc88d05b8746d9f327d P 05f18aba3ea94033ad3641fce781f708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66a65d1a2ec44c79932a15778993706" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 36923 } } peers { permanent_uuid: "fc6f41614da44d429522b576d0ff0187" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 44989 } } peers { permanent_uuid: "05f18aba3ea94033ad3641fce781f708" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 38589 } }
I20260810 22:36:32.719494 32292 raft_consensus.cc:374] T 7c6fe54437264fc88d05b8746d9f327d P 05f18aba3ea94033ad3641fce781f708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:36:32.719617 32292 raft_consensus.cc:729] T 7c6fe54437264fc88d05b8746d9f327d P 05f18aba3ea94033ad3641fce781f708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f18aba3ea94033ad3641fce781f708, State: Initialized, Role: FOLLOWER
I20260810 22:36:32.720247 32292 consensus_queue.cc:260] T 7c6fe54437264fc88d05b8746d9f327d P 05f18aba3ea94033ad3641fce781f708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66a65d1a2ec44c79932a15778993706" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 36923 } } peers { permanent_uuid: "fc6f41614da44d429522b576d0ff0187" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 44989 } } peers { permanent_uuid: "05f18aba3ea94033ad3641fce781f708" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 38589 } }
I20260810 22:36:32.721371 32099 heartbeater.cc:503] Master 127.16.233.126:40771 was elected leader, sending a full tablet report...
I20260810 22:36:32.725390 32292 ts_tablet_manager.cc:1428] T 7c6fe54437264fc88d05b8746d9f327d P 05f18aba3ea94033ad3641fce781f708: Time spent starting tablet: real 0.024s user 0.004s sys 0.009s
W20260810 22:36:32.726392 31948 tablet.cc:2369] T 7c6fe54437264fc88d05b8746d9f327d P fc6f41614da44d429522b576d0ff0187: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 22:36:32.768110 32100 tablet.cc:2369] T 7c6fe54437264fc88d05b8746d9f327d P 05f18aba3ea94033ad3641fce781f708: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 10 22:36:32 dist-test-slave-77v9 krb5kdc[30565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401391, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.126@KRBTEST.COM for HTTP/127.16.233.84@KRBTEST.COM
W20260810 22:36:32.804352 32254 tablet.cc:2369] T 7c6fe54437264fc88d05b8746d9f327d P f66a65d1a2ec44c79932a15778993706: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:36:32.829916 32298 raft_consensus.cc:482] T 7c6fe54437264fc88d05b8746d9f327d P 05f18aba3ea94033ad3641fce781f708 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:36:32.830391 32298 raft_consensus.cc:504] T 7c6fe54437264fc88d05b8746d9f327d P 05f18aba3ea94033ad3641fce781f708 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66a65d1a2ec44c79932a15778993706" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 36923 } } peers { permanent_uuid: "fc6f41614da44d429522b576d0ff0187" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 44989 } } peers { permanent_uuid: "05f18aba3ea94033ad3641fce781f708" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 38589 } }
I20260810 22:36:32.832001 32298 leader_election.cc:288] T 7c6fe54437264fc88d05b8746d9f327d P 05f18aba3ea94033ad3641fce781f708 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f66a65d1a2ec44c79932a15778993706 (127.16.233.67:36923), fc6f41614da44d429522b576d0ff0187 (127.16.233.65:44989)
I20260810 22:36:32.841543 32285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:32.833449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:51779 (local address 127.16.233.67:36923)
0810 22:36:32.833581 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 22:36:32.833587 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:36:32.833604 (+ 17us) server_negotiation.cc:407] Connection header received
0810 22:36:32.833665 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:36:32.833671 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:36:32.833729 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:36:32.833874 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:36:32.834522 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:32.835420 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:32.839403 (+ 3983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:32.840061 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:32.840177 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:32.840234 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 22:36:32.841287 (+ 1053us) server_negotiation.cc:299] Negotiation successful
0810 22:36:32.841386 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 22:36:32.842366 32289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:32.833610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:48793 (local address 127.16.233.65:44989)
0810 22:36:32.833797 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 22:36:32.833802 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:36:32.834772 (+ 970us) server_negotiation.cc:407] Connection header received
0810 22:36:32.834974 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:36:32.834978 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:36:32.835026 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:36:32.835076 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:36:32.835541 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:32.836396 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:32.840194 (+ 3798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:32.840855 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:32.840962 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:32.841009 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0810 22:36:32.842152 (+ 1143us) server_negotiation.cc:299] Negotiation successful
0810 22:36:32.842243 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260810 22:36:32.843281 32303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:32.832264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.67:36923 (local address 127.16.233.66:51779)
0810 22:36:32.833310 (+ 1046us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:32.833330 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 22:36:32.833486 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:36:32.833885 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:36:32.833889 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:36:32.833912 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:36:32.834343 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:32.834353 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:32.835788 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:32.835791 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:32.838159 (+ 2368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:32.838171 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:32.840894 (+ 2723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:32.840897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:32.841120 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:32.841147 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 22:36:32.842783 (+ 1636us) client_negotiation.cc:240] Negotiation successful
0810 22:36:32.842876 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":938,"thread_start_us":170,"threads_started":1}
I20260810 22:36:32.844484 32302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:32.832799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.65:44989 (local address 127.16.233.66:48793)
0810 22:36:32.834676 (+ 1877us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:32.834693 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 22:36:32.834849 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:36:32.835224 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:36:32.835227 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:36:32.835245 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:36:32.835395 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:32.835402 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:32.837139 (+ 1737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:32.837143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:32.840019 (+ 2876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:32.840033 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:32.841890 (+ 1857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:32.841894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:32.842002 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:32.842025 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 22:36:32.844226 (+ 2201us) client_negotiation.cc:240] Negotiation successful
0810 22:36:32.844322 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1801,"thread_start_us":104,"threads_started":1}
I20260810 22:36:32.844411 32188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c6fe54437264fc88d05b8746d9f327d" candidate_uuid: "05f18aba3ea94033ad3641fce781f708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f66a65d1a2ec44c79932a15778993706" is_pre_election: true
I20260810 22:36:32.844775 32188 raft_consensus.cc:2454] T 7c6fe54437264fc88d05b8746d9f327d P f66a65d1a2ec44c79932a15778993706 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05f18aba3ea94033ad3641fce781f708 in term 0.
I20260810 22:36:32.845362 31894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c6fe54437264fc88d05b8746d9f327d" candidate_uuid: "05f18aba3ea94033ad3641fce781f708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc6f41614da44d429522b576d0ff0187" is_pre_election: true
I20260810 22:36:32.845490 31974 leader_election.cc:302] T 7c6fe54437264fc88d05b8746d9f327d P 05f18aba3ea94033ad3641fce781f708 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f18aba3ea94033ad3641fce781f708, f66a65d1a2ec44c79932a15778993706; no voters:
I20260810 22:36:32.845647 31894 raft_consensus.cc:2454] T 7c6fe54437264fc88d05b8746d9f327d P fc6f41614da44d429522b576d0ff0187 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05f18aba3ea94033ad3641fce781f708 in term 0.
I20260810 22:36:32.845834 32298 raft_consensus.cc:2781] T 7c6fe54437264fc88d05b8746d9f327d P 05f18aba3ea94033ad3641fce781f708 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:36:32.845911 32298 raft_consensus.cc:482] T 7c6fe54437264fc88d05b8746d9f327d P 05f18aba3ea94033ad3641fce781f708 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:36:32.845963 32298 raft_consensus.cc:3037] T 7c6fe54437264fc88d05b8746d9f327d P 05f18aba3ea94033ad3641fce781f708 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:36:32.848120 32298 raft_consensus.cc:504] T 7c6fe54437264fc88d05b8746d9f327d P 05f18aba3ea94033ad3641fce781f708 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66a65d1a2ec44c79932a15778993706" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 36923 } } peers { permanent_uuid: "fc6f41614da44d429522b576d0ff0187" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 44989 } } peers { permanent_uuid: "05f18aba3ea94033ad3641fce781f708" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 38589 } }
I20260810 22:36:32.848559 32298 leader_election.cc:288] T 7c6fe54437264fc88d05b8746d9f327d P 05f18aba3ea94033ad3641fce781f708 [CANDIDATE]: Term 1 election: Requested vote from peers f66a65d1a2ec44c79932a15778993706 (127.16.233.67:36923), fc6f41614da44d429522b576d0ff0187 (127.16.233.65:44989)
I20260810 22:36:32.849510 31894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c6fe54437264fc88d05b8746d9f327d" candidate_uuid: "05f18aba3ea94033ad3641fce781f708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc6f41614da44d429522b576d0ff0187"
I20260810 22:36:32.849687 31894 raft_consensus.cc:3037] T 7c6fe54437264fc88d05b8746d9f327d P fc6f41614da44d429522b576d0ff0187 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:36:32.850152 32188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c6fe54437264fc88d05b8746d9f327d" candidate_uuid: "05f18aba3ea94033ad3641fce781f708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f66a65d1a2ec44c79932a15778993706"
I20260810 22:36:32.850320 32188 raft_consensus.cc:3037] T 7c6fe54437264fc88d05b8746d9f327d P f66a65d1a2ec44c79932a15778993706 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:36:32.851120 31894 raft_consensus.cc:2454] T 7c6fe54437264fc88d05b8746d9f327d P fc6f41614da44d429522b576d0ff0187 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05f18aba3ea94033ad3641fce781f708 in term 1.
I20260810 22:36:32.851565 31971 leader_election.cc:302] T 7c6fe54437264fc88d05b8746d9f327d P 05f18aba3ea94033ad3641fce781f708 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f18aba3ea94033ad3641fce781f708, fc6f41614da44d429522b576d0ff0187; no voters:
I20260810 22:36:32.851928 32298 raft_consensus.cc:2781] T 7c6fe54437264fc88d05b8746d9f327d P 05f18aba3ea94033ad3641fce781f708 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:36:32.852267 32298 raft_consensus.cc:686] T 7c6fe54437264fc88d05b8746d9f327d P 05f18aba3ea94033ad3641fce781f708 [term 1 LEADER]: Becoming Leader. State: Replica: 05f18aba3ea94033ad3641fce781f708, State: Running, Role: LEADER
I20260810 22:36:32.852314 32188 raft_consensus.cc:2454] T 7c6fe54437264fc88d05b8746d9f327d P f66a65d1a2ec44c79932a15778993706 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05f18aba3ea94033ad3641fce781f708 in term 1.
I20260810 22:36:32.852690 32298 consensus_queue.cc:237] T 7c6fe54437264fc88d05b8746d9f327d P 05f18aba3ea94033ad3641fce781f708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66a65d1a2ec44c79932a15778993706" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 36923 } } peers { permanent_uuid: "fc6f41614da44d429522b576d0ff0187" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 44989 } } peers { permanent_uuid: "05f18aba3ea94033ad3641fce781f708" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 38589 } }
I20260810 22:36:32.856967 31724 catalog_manager.cc:5495] T 7c6fe54437264fc88d05b8746d9f327d P 05f18aba3ea94033ad3641fce781f708 reported cstate change: term changed from 0 to 1, leader changed from <none> to 05f18aba3ea94033ad3641fce781f708 (127.16.233.66). New cstate: current_term: 1 leader_uuid: "05f18aba3ea94033ad3641fce781f708" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f66a65d1a2ec44c79932a15778993706" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 36923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc6f41614da44d429522b576d0ff0187" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 44989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05f18aba3ea94033ad3641fce781f708" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 38589 } health_report { overall_health: HEALTHY } } }
I20260810 22:36:32.884284 32288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:32.878383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48986 (local address 127.16.233.66:38589)
0810 22:36:32.878541 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0810 22:36:32.878546 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:36:32.878565 (+ 19us) server_negotiation.cc:407] Connection header received
0810 22:36:32.878673 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:36:32.878677 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:36:32.878749 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:36:32.878883 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:36:32.879212 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:32.880057 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:32.881485 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:32.881826 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:32.881954 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:32.882512 (+ 558us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:36:32.883185 (+ 673us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:36:32.883330 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 22:36:32.883890 (+ 560us) server_negotiation.cc:299] Negotiation successful
0810 22:36:32.883967 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 22:36:32.893923 31894 raft_consensus.cc:1261] T 7c6fe54437264fc88d05b8746d9f327d P fc6f41614da44d429522b576d0ff0187 [term 1 FOLLOWER]: Refusing update from remote peer 05f18aba3ea94033ad3641fce781f708: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:36:32.895166 32188 raft_consensus.cc:1261] T 7c6fe54437264fc88d05b8746d9f327d P f66a65d1a2ec44c79932a15778993706 [term 1 FOLLOWER]: Refusing update from remote peer 05f18aba3ea94033ad3641fce781f708: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:36:32.895671 32298 consensus_queue.cc:1035] T 7c6fe54437264fc88d05b8746d9f327d P 05f18aba3ea94033ad3641fce781f708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc6f41614da44d429522b576d0ff0187" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 44989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:36:32.896164 32304 consensus_queue.cc:1035] T 7c6fe54437264fc88d05b8746d9f327d P 05f18aba3ea94033ad3641fce781f708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f66a65d1a2ec44c79932a15778993706" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 36923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:36:32.913280 32308 mvcc.cc:205] Tried to move back new op lower bound from 7317100105281495040 to 7317100105128292352. Current Snapshot: MvccSnapshot[applied={T|T < 7317100105281495040}]
I20260810 22:36:32.915858 32309 mvcc.cc:205] Tried to move back new op lower bound from 7317100105281495040 to 7317100105128292352. Current Snapshot: MvccSnapshot[applied={T|T < 7317100105281495040}]
I20260810 22:36:32.924690 31725 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:36:32.924938 31725 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:36:32.927910 31725 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1176e1761df84e2b94862e38dd0a8d36: Sending DeleteTablet for 3 replicas of tablet 7c6fe54437264fc88d05b8746d9f327d
I20260810 22:36:32.930382 31868 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c6fe54437264fc88d05b8746d9f327d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:36:32 UTC) from {username='kudu', principal='kudu/127.16.233.126@KRBTEST.COM'} at 127.0.0.1:44108
I20260810 22:36:32.931457 32022 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c6fe54437264fc88d05b8746d9f327d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:36:32 UTC) from {username='kudu', principal='kudu/127.16.233.126@KRBTEST.COM'} at 127.0.0.1:48972
I20260810 22:36:32.931875 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 31767
I20260810 22:36:32.934959 32154 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c6fe54437264fc88d05b8746d9f327d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:36:32 UTC) from {username='kudu', principal='kudu/127.16.233.126@KRBTEST.COM'} at 127.0.0.1:52448
I20260810 22:36:32.937330 32321 tablet_replica.cc:330] T 7c6fe54437264fc88d05b8746d9f327d P f66a65d1a2ec44c79932a15778993706: stopping tablet replica
I20260810 22:36:32.937896 32321 raft_consensus.cc:2229] T 7c6fe54437264fc88d05b8746d9f327d P f66a65d1a2ec44c79932a15778993706 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:36:32.938445 32321 raft_consensus.cc:2258] T 7c6fe54437264fc88d05b8746d9f327d P f66a65d1a2ec44c79932a15778993706 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:36:32.941072 32321 ts_tablet_manager.cc:1905] T 7c6fe54437264fc88d05b8746d9f327d P f66a65d1a2ec44c79932a15778993706: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:36:32.944692 32321 ts_tablet_manager.cc:1918] T 7c6fe54437264fc88d05b8746d9f327d P f66a65d1a2ec44c79932a15778993706: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:36:32.945055 32321 log.cc:1196] T 7c6fe54437264fc88d05b8746d9f327d P f66a65d1a2ec44c79932a15778993706: Deleting WAL directory at /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-2/wal/wals/7c6fe54437264fc88d05b8746d9f327d
I20260810 22:36:32.945641 32321 ts_tablet_manager.cc:1939] T 7c6fe54437264fc88d05b8746d9f327d P f66a65d1a2ec44c79932a15778993706: Deleting consensus metadata
I20260810 22:36:32.947547 31711 catalog_manager.cc:4841] TS f66a65d1a2ec44c79932a15778993706 (127.16.233.67:36923): tablet 7c6fe54437264fc88d05b8746d9f327d (table test-table [id=75d4d210da684135b4d1bd5b9f7d13e1]) successfully deleted
I20260810 22:36:32.950037 32320 tablet_replica.cc:330] T 7c6fe54437264fc88d05b8746d9f327d P 05f18aba3ea94033ad3641fce781f708: stopping tablet replica
I20260810 22:36:32.950496 32320 raft_consensus.cc:2229] T 7c6fe54437264fc88d05b8746d9f327d P 05f18aba3ea94033ad3641fce781f708 [term 1 LEADER]: Raft consensus shutting down.
I20260810 22:36:32.956853 32320 raft_consensus.cc:2258] T 7c6fe54437264fc88d05b8746d9f327d P 05f18aba3ea94033ad3641fce781f708 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:36:32.959081 32320 ts_tablet_manager.cc:1905] T 7c6fe54437264fc88d05b8746d9f327d P 05f18aba3ea94033ad3641fce781f708: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:36:32.962658 32320 ts_tablet_manager.cc:1918] T 7c6fe54437264fc88d05b8746d9f327d P 05f18aba3ea94033ad3641fce781f708: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:36:32.962797 32320 log.cc:1196] T 7c6fe54437264fc88d05b8746d9f327d P 05f18aba3ea94033ad3641fce781f708: Deleting WAL directory at /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786401215402979-17317-0/minicluster-data/ts-1/wal/wals/7c6fe54437264fc88d05b8746d9f327d
I20260810 22:36:32.963198 32320 ts_tablet_manager.cc:1939] T 7c6fe54437264fc88d05b8746d9f327d P 05f18aba3ea94033ad3641fce781f708: Deleting consensus metadata
I20260810 22:36:32.964680 31709 catalog_manager.cc:4841] TS 05f18aba3ea94033ad3641fce781f708 (127.16.233.66:38589): tablet 7c6fe54437264fc88d05b8746d9f327d (table test-table [id=75d4d210da684135b4d1bd5b9f7d13e1]) successfully deleted
W20260810 22:36:32.965250 31708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.233.65:44989 (error 108)
I20260810 22:36:32.966341 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 31951
I20260810 22:36:32.982030 32286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:32.981596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.65:44989 (local address 127.0.0.1:44112)
0810 22:36:32.981827 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:32.981921 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.65:44989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
W20260810 22:36:32.982276 31708 catalog_manager.cc:4584] TS fc6f41614da44d429522b576d0ff0187 (127.16.233.65:44989): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7c6fe54437264fc88d05b8746d9f327d: Network error: Client connection negotiation failed: client connection to 127.16.233.65:44989: connect: Connection refused (error 111)
I20260810 22:36:32.986356 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 32102
I20260810 22:36:32.996737 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 31690
I20260810 22:36:33.008801 17317 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260810 22:36:33.355754 17317 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260810 22:36:33.355948 17317 mini_ranger.cc:66] Stopping Ranger...
I20260810 22:36:33.702932 17317 mini_ranger.cc:68] Stopped Ranger
2026-08-10 22:36:33.703 UTC [30605] LOG: received smart shutdown request
2026-08-10 22:36:33.708 UTC [30605] LOG: background worker "logical replication launcher" (PID 30614) exited with exit code 1
2026-08-10 22:36:33.709 UTC [30609] LOG: shutting down
2026-08-10 22:36:33.936 UTC [30605] LOG: database system is shut down
2026-08-10T22:36:33Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (53677 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:36:34 dist-test-slave-77v9 krb5kdc[32332](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:36:34 dist-test-slave-77v9 krb5kdc[32332](info): set up 2 sockets
Aug 10 22:36:34 dist-test-slave-77v9 krb5kdc[32332](info): commencing operation
krb5kdc: starting...
W20260810 22:36:35.252244 17317 mini_kdc.cc:121] Time spent starting KDC: real 1.146s user 0.002s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Aug 10 22:36:35 dist-test-slave-77v9 krb5kdc[32332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401395, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:36:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:36:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.233.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab.
Entry for principal kudu/127.16.233.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab.
I20260810 22:36:35.402133 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:44231
--webserver_interface=127.16.233.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab
--principal=kudu/127.16.233.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:36035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.126:44231,127.16.233.125:36301,127.16.233.124:46485
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:36:35.530807 32350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:36:35.531211 32350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:36:35.531309 32350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:36:35.535763 32350 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:36:35.535897 32350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:36:35.535935 32350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:36:35.535977 32350 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:36:35.536034 32350 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:36:35.542053 32350 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:36035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.126:44231,127.16.233.125:36301,127.16.233.124:46485
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab
--principal=kudu/127.16.233.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:44231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:36:35.543417 32350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:36:35.544555 32350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:36:35.554399 32356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:36:35.554402 32358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:36:35.554543 32355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:36:35.554767 32350 server_base.cc:1034] running on GCE node
I20260810 22:36:35.555465 32350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:36:35.556759 32350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:36:35.558007 32350 hybrid_clock.cc:648] HybridClock initialized: now 1786401395557976 us; error 50 us; skew 500 ppm
Aug 10 22:36:35 dist-test-slave-77v9 krb5kdc[32332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401395, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:36:35.561288 32350 init.cc:377] Logged in from keytab as kudu/127.16.233.126@KRBTEST.COM (short username kudu)
I20260810 22:36:35.562278 32350 webserver.cc:460] Webserver started at http://127.16.233.126:33691/ using document root <none> and password file <none>
I20260810 22:36:35.562851 32350 fs_manager.cc:359] Metadata directory not provided
I20260810 22:36:35.562963 32350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:36:35.563213 32350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:36:35.565568 32350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/minicluster-data/master-0/data/instance:
uuid: "ea876dec6927461fb6e989b21abf5331"
format_stamp: "Formatted at 2026-08-10 22:36:35 on dist-test-slave-77v9"
server_key: "b9adbb3cb0ce8b0b6cf2a6a192eb8dc5"
server_key_iv: "dd1b32a06dba9a6bc8110d39ef88f5a1"
server_key_version: "encryptionkey@0"
I20260810 22:36:35.566267 32350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/minicluster-data/master-0/wal/instance:
uuid: "ea876dec6927461fb6e989b21abf5331"
format_stamp: "Formatted at 2026-08-10 22:36:35 on dist-test-slave-77v9"
server_key: "b9adbb3cb0ce8b0b6cf2a6a192eb8dc5"
server_key_iv: "dd1b32a06dba9a6bc8110d39ef88f5a1"
server_key_version: "encryptionkey@0"
I20260810 22:36:35.570863 32350 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 22:36:35.573935 32366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:36:35.575387 32350 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:36:35.575589 32350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/minicluster-data/master-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/minicluster-data/master-0/wal
uuid: "ea876dec6927461fb6e989b21abf5331"
format_stamp: "Formatted at 2026-08-10 22:36:35 on dist-test-slave-77v9"
server_key: "b9adbb3cb0ce8b0b6cf2a6a192eb8dc5"
server_key_iv: "dd1b32a06dba9a6bc8110d39ef88f5a1"
server_key_version: "encryptionkey@0"
I20260810 22:36:35.575747 32350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:36:35.610612 32350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:36:35.614552 32350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:36:35.614825 32350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:36:35.623250 32350 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.126:44231
I20260810 22:36:35.623593 32423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.126:44231 every 8 connection(s)
I20260810 22:36:35.624523 32350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
I20260810 22:36:35.627946 32424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:36:35.631945 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 32350
I20260810 22:36:35.632134 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/minicluster-data/master-0/wal/instance
I20260810 22:36:35.632450 17317 external_mini_cluster.cc:1428] Setting key 938791169ae4a12146d88c8bb8c1a7ef
I20260810 22:36:35.632795 32424 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.233.126" port: 44231 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.16.233.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.125@KRBTEST.COM" created.
Aug 10 22:36:35 dist-test-slave-77v9 krb5kdc[32332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401395, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.126@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:36:35.654474 32428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:35.636283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:44231 (local address 127.0.0.1:33658)
0810 22:36:35.636993 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:35.637062 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0810 22:36:35.638490 (+ 1428us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:36:35.639411 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:36:35.639420 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:36:35.639998 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:36:35.640560 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:35.640589 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:35.641977 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:35.641983 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:35.643318 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:35.643334 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:35.644108 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:35.644116 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:35.644264 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:35.644960 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:36:35.644978 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:36:35.648261 (+ 3283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:36:35.650699 (+ 2438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:36:35.650711 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:36:35.650725 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:36:35.651079 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:36:35.651563 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:36:35.651566 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:36:35.651568 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:36:35.651701 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:36:35.652646 (+ 945us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:36:35.652653 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:36:35.652809 (+ 156us) client_negotiation.cc:769] Sending connection context
0810 22:36:35.653123 (+ 314us) client_negotiation.cc:240] Negotiation successful
0810 22:36:35.653550 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"mutex_wait_us":1,"thread_start_us":184,"threads_started":1}
I20260810 22:36:35.654484 32429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:35.636283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33658 (local address 127.16.233.126:44231)
0810 22:36:35.636970 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0810 22:36:35.636989 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 22:36:35.637203 (+ 214us) server_negotiation.cc:407] Connection header received
0810 22:36:35.638933 (+ 1730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:36:35.638939 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:36:35.639162 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:36:35.639334 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:36:35.640775 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:35.641780 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:35.643477 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:35.643871 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:35.644071 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:35.648492 (+ 4421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:36:35.648499 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:36:35.648520 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:36:35.648551 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:36:35.650544 (+ 1993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:36:35.651212 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:36:35.651221 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:36:35.651226 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:36:35.651388 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:36:35.651816 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:36:35.651822 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:36:35.651827 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:36:35.652478 (+ 651us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:36:35.652666 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0810 22:36:35.653226 (+ 560us) server_negotiation.cc:299] Negotiation successful
0810 22:36:35.653657 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":154,"server-negotiator.queue_time_us":550,"spinlock_wait_cycles":57600,"thread_start_us":184,"threads_started":1}
I20260810 22:36:35.657231 32424 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.233.125" port: 36301 } has no permanent_uuid. Determining permanent_uuid...
I20260810 22:36:35.658154 32428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:35.657807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.125:36301 (local address 127.0.0.1:39024)
0810 22:36:35.658001 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:35.658079 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.125:36301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260810 22:36:35.658540 32370 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.233.125:36301: connect: Connection refused (error 111)
I20260810 22:36:35.659598 32428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:35.659259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.125:36301 (local address 127.0.0.1:39032)
0810 22:36:35.659382 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:35.659434 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.125:36301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260810 22:36:35.659792 32424 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.233.125:36301: Network error: Client connection negotiation failed: client connection to 127.16.233.125:36301: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.125.keytab.
Entry for principal kudu/127.16.233.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.125.keytab.
I20260810 22:36:35.669063 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.125:36301
--webserver_interface=127.16.233.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.125.keytab
--principal=kudu/127.16.233.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:36035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.126:44231,127.16.233.125:36301,127.16.233.124:46485
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 22:36:35.709138 32424 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.233.125" port: 36301 } attempt: 1
I20260810 22:36:35.710229 32428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:35.709834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.125:36301 (local address 127.0.0.1:39036)
0810 22:36:35.710059 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:35.710141 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.125:36301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260810 22:36:35.711190 32428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:35.710923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.125:36301 (local address 127.0.0.1:39040)
0810 22:36:35.711059 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:35.711119 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.125:36301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260810 22:36:35.711344 32424 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.233.125:36301: Network error: Client connection negotiation failed: client connection to 127.16.233.125:36301: connect: Connection refused (error 111)
I20260810 22:36:35.779569 32424 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.233.125" port: 36301 } attempt: 2
I20260810 22:36:35.780646 32428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:35.780275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.125:36301 (local address 127.0.0.1:39042)
0810 22:36:35.780472 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:35.780556 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.125:36301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260810 22:36:35.781667 32428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:35.781399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.125:36301 (local address 127.0.0.1:39058)
0810 22:36:35.781538 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:35.781596 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.125:36301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260810 22:36:35.781865 32424 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.233.125:36301: Network error: Client connection negotiation failed: client connection to 127.16.233.125:36301: connect: Connection refused (error 111)
W20260810 22:36:35.797581 32435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:36:35.797914 32435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:36:35.797993 32435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:36:35.802503 32435 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:36:35.802600 32435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:36:35.802620 32435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:36:35.802639 32435 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:36:35.802655 32435 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:36:35.808766 32435 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:36035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.126:44231,127.16.233.125:36301,127.16.233.124:46485
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.125.keytab
--principal=kudu/127.16.233.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.125:36301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.233.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:36:35.810158 32435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:36:35.811329 32435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:36:35.821277 32441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:36:35.821311 32443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:36:35.821277 32440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:36:35.821535 32435 server_base.cc:1034] running on GCE node
I20260810 22:36:35.822412 32435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:36:35.823843 32435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:36:35.825060 32435 hybrid_clock.cc:648] HybridClock initialized: now 1786401395825027 us; error 48 us; skew 500 ppm
Aug 10 22:36:35 dist-test-slave-77v9 krb5kdc[32332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401395, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:36:35.828574 32435 init.cc:377] Logged in from keytab as kudu/127.16.233.125@KRBTEST.COM (short username kudu)
I20260810 22:36:35.829685 32435 webserver.cc:460] Webserver started at http://127.16.233.125:33315/ using document root <none> and password file <none>
I20260810 22:36:35.830332 32435 fs_manager.cc:359] Metadata directory not provided
I20260810 22:36:35.830406 32435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:36:35.830596 32435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:36:35.832948 32435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/minicluster-data/master-1/data/instance:
uuid: "60255f340e9a41049e72acdde7dff5cb"
format_stamp: "Formatted at 2026-08-10 22:36:35 on dist-test-slave-77v9"
server_key: "da742909fd2e6f6bc71ff21065806f30"
server_key_iv: "67029bda19d282b8093d360bc2f87f73"
server_key_version: "encryptionkey@0"
I20260810 22:36:35.833588 32435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/minicluster-data/master-1/wal/instance:
uuid: "60255f340e9a41049e72acdde7dff5cb"
format_stamp: "Formatted at 2026-08-10 22:36:35 on dist-test-slave-77v9"
server_key: "da742909fd2e6f6bc71ff21065806f30"
server_key_iv: "67029bda19d282b8093d360bc2f87f73"
server_key_version: "encryptionkey@0"
I20260810 22:36:35.838109 32435 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260810 22:36:35.841195 32451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:36:35.842900 32435 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260810 22:36:35.843051 32435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/minicluster-data/master-1/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/minicluster-data/master-1/wal
uuid: "60255f340e9a41049e72acdde7dff5cb"
format_stamp: "Formatted at 2026-08-10 22:36:35 on dist-test-slave-77v9"
server_key: "da742909fd2e6f6bc71ff21065806f30"
server_key_iv: "67029bda19d282b8093d360bc2f87f73"
server_key_version: "encryptionkey@0"
I20260810 22:36:35.843175 32435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:36:35.859737 32435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:36:35.863432 32435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:36:35.863691 32435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:36:35.871781 32435 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.125:36301
I20260810 22:36:35.872224 32510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.125:36301 every 8 connection(s)
I20260810 22:36:35.872954 32435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/minicluster-data/master-1/data/info.pb
I20260810 22:36:35.873137 32424 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.233.125" port: 36301 } attempt: 3
I20260810 22:36:35.876078 32511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:36:35.878199 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 32435
I20260810 22:36:35.878374 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/minicluster-data/master-1/wal/instance
I20260810 22:36:35.878710 17317 external_mini_cluster.cc:1428] Setting key f05e0323d7044541ed35d83a4faa451a
I20260810 22:36:35.881095 32511 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.233.126" port: 44231 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 22:36:35 dist-test-slave-77v9 krb5kdc[32332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401395, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.126@KRBTEST.COM for kudu/127.16.233.125@KRBTEST.COM
I20260810 22:36:35.888870 32428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:35.873779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.125:36301 (local address 127.0.0.1:39066)
0810 22:36:35.873964 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:35.873983 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 22:36:35.874094 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:36:35.876520 (+ 2426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:36:35.876524 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:36:35.876742 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:36:35.876945 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:35.876954 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:35.878602 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:35.878606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:35.879456 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:35.879470 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:35.880632 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:35.880637 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:35.880753 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:35.881469 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:36:35.881486 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:36:35.883251 (+ 1765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:36:35.886331 (+ 3080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:36:35.886334 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:36:35.886336 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:36:35.886639 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:36:35.887100 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:36:35.887103 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:36:35.887105 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:36:35.887223 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:36:35.888191 (+ 968us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:36:35.888195 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:36:35.888325 (+ 130us) client_negotiation.cc:769] Sending connection context
0810 22:36:35.888425 (+ 100us) client_negotiation.cc:240] Negotiation successful
0810 22:36:35.888601 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
WARNING: no policy specified for kudu/127.16.233.124@KRBTEST.COM; defaulting to no policy
Aug 10 22:36:35 dist-test-slave-77v9 krb5kdc[32332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401395, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.125@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:36:35.890551 32513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:35.874591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39066 (local address 127.16.233.125:36301)
0810 22:36:35.875065 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0810 22:36:35.875084 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 22:36:35.875134 (+ 50us) server_negotiation.cc:407] Connection header received
0810 22:36:35.875799 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:36:35.875816 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:36:35.876183 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:36:35.876514 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:36:35.877103 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:35.878368 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:35.879641 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:35.880464 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:35.880746 (+ 282us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:35.883522 (+ 2776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:36:35.883531 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:36:35.883551 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:36:35.883593 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:36:35.886189 (+ 2596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:36:35.886784 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:36:35.886795 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:36:35.886805 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:36:35.886986 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:36:35.887335 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:36:35.887339 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:36:35.887342 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:36:35.888016 (+ 674us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:36:35.888957 (+ 941us) server_negotiation.cc:1035] Waiting for connection context
0810 22:36:35.889146 (+ 189us) server_negotiation.cc:299] Negotiation successful
0810 22:36:35.889496 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":194,"threads_started":1}
I20260810 22:36:35.893245 32424 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.233.124" port: 46485 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.124@KRBTEST.COM" created.
I20260810 22:36:35.894115 32428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:35.893848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.124:46485 (local address 127.0.0.1:33944)
0810 22:36:35.893981 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:35.894037 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.124:46485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260810 22:36:35.895170 32428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:35.894916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.124:46485 (local address 127.0.0.1:33960)
0810 22:36:35.895061 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:35.895116 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.124:46485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260810 22:36:35.895411 32429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:35.883307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33666 (local address 127.16.233.126:44231)
0810 22:36:35.883452 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 22:36:35.883459 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 22:36:35.883952 (+ 493us) server_negotiation.cc:407] Connection header received
0810 22:36:35.884150 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:36:35.884153 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:36:35.884208 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:36:35.884263 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:36:35.885266 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:35.885813 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:35.886957 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:35.887953 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:35.888106 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:35.890959 (+ 2853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:36:35.890963 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:36:35.890966 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:36:35.891003 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:36:35.892745 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:36:35.893568 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:36:35.893572 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:36:35.893574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:36:35.893715 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:36:35.894195 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:36:35.894200 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:36:35.894202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:36:35.894436 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:36:35.894515 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 22:36:35.895013 (+ 498us) server_negotiation.cc:299] Negotiation successful
0810 22:36:35.895171 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260810 22:36:35.895547 32424 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.233.124:46485: Network error: Client connection negotiation failed: client connection to 127.16.233.124:46485: connect: Connection refused (error 111)
I20260810 22:36:35.896191 32515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:35.883302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:44231 (local address 127.0.0.1:33666)
0810 22:36:35.883818 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:35.883874 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 22:36:35.884043 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:36:35.884416 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:36:35.884420 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:36:35.884951 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:36:35.885131 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:35.885138 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:35.885950 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:35.885955 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:35.886839 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:35.886849 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:35.888124 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:35.888128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:35.888269 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:35.888895 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:36:35.888912 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:36:35.890779 (+ 1867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:36:35.893005 (+ 2226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:36:35.893017 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:36:35.893026 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:36:35.893470 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:36:35.893940 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:36:35.893946 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:36:35.893951 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:36:35.894073 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:36:35.894553 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:36:35.894562 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:36:35.894732 (+ 170us) client_negotiation.cc:769] Sending connection context
0810 22:36:35.895626 (+ 894us) client_negotiation.cc:240] Negotiation successful
0810 22:36:35.895889 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":236,"threads_started":1}
I20260810 22:36:35.897589 32511 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.233.125" port: 36301 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 22:36:35 dist-test-slave-77v9 krb5kdc[32332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401395, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.125@KRBTEST.COM for kudu/127.16.233.125@KRBTEST.COM
I20260810 22:36:35.908159 32515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:35.898268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.125:36301 (local address 127.0.0.1:39072)
0810 22:36:35.898530 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:35.898552 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 22:36:35.898666 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:36:35.898907 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:36:35.898913 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:36:35.899139 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:36:35.899317 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:35.899328 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:35.900080 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:35.900087 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:35.901112 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:35.901129 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:35.901687 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:35.901692 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:35.901835 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:35.902412 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:36:35.902429 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:36:35.904061 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:36:35.906117 (+ 2056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:36:35.906122 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:36:35.906128 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:36:35.906355 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:36:35.906842 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:36:35.906845 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:36:35.906846 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:36:35.906944 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:36:35.907545 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:36:35.907548 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:36:35.907641 (+ 93us) client_negotiation.cc:769] Sending connection context
0810 22:36:35.907754 (+ 113us) client_negotiation.cc:240] Negotiation successful
0810 22:36:35.907963 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260810 22:36:35.908159 32513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:35.898322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39072 (local address 127.16.233.125:36301)
0810 22:36:35.898443 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0810 22:36:35.898448 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:36:35.898626 (+ 178us) server_negotiation.cc:407] Connection header received
0810 22:36:35.898752 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:36:35.898754 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:36:35.898794 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:36:35.898862 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:36:35.899460 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:35.899980 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:35.901277 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:35.901542 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:35.901639 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:35.904276 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:36:35.904280 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:36:35.904282 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:36:35.904308 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:36:35.905960 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:36:35.906515 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:36:35.906518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:36:35.906519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:36:35.906674 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:36:35.907078 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:36:35.907081 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:36:35.907082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:36:35.907430 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:36:35.907545 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 22:36:35.907812 (+ 267us) server_negotiation.cc:299] Negotiation successful
0810 22:36:35.908043 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":104,"server-negotiator.queue_time_us":43}
I20260810 22:36:35.909227 32511 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.233.124" port: 46485 } has no permanent_uuid. Determining permanent_uuid...
I20260810 22:36:35.910149 32515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:35.909786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.124:46485 (local address 127.0.0.1:33972)
0810 22:36:35.909985 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:35.910079 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.124:46485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260810 22:36:35.910791 32452 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.233.124:46485: connect: Connection refused (error 111)
I20260810 22:36:35.911831 32515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:35.911589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.124:46485 (local address 127.0.0.1:33980)
0810 22:36:35.911711 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:35.911767 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.124:46485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260810 22:36:35.912001 32511 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.233.124:46485: Network error: Client connection negotiation failed: client connection to 127.16.233.124:46485: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.124.keytab.
Entry for principal kudu/127.16.233.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.124.keytab.
I20260810 22:36:35.916575 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.124:46485
--webserver_interface=127.16.233.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.124.keytab
--principal=kudu/127.16.233.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:36035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.126:44231,127.16.233.125:36301,127.16.233.124:46485
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 22:36:35.926738 32424 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.233.124" port: 46485 } attempt: 1
I20260810 22:36:35.927744 32428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:35.927420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.124:46485 (local address 127.0.0.1:33982)
0810 22:36:35.927583 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:35.927658 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.124:46485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260810 22:36:35.928571 32428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:35.928331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.124:46485 (local address 127.0.0.1:33998)
0810 22:36:35.928449 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:35.928503 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.124:46485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 22:36:35.928785 32424 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.233.124:46485: Network error: Client connection negotiation failed: client connection to 127.16.233.124:46485: connect: Connection refused (error 111)
I20260810 22:36:35.961342 32511 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.233.124" port: 46485 } attempt: 1
I20260810 22:36:35.962343 32515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:35.962019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.124:46485 (local address 127.0.0.1:34006)
0810 22:36:35.962182 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:35.962253 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.124:46485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260810 22:36:35.963295 32515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:35.963047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.124:46485 (local address 127.0.0.1:34018)
0810 22:36:35.963171 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:35.963230 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.124:46485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260810 22:36:35.963487 32511 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.233.124:46485: Network error: Client connection negotiation failed: client connection to 127.16.233.124:46485: connect: Connection refused (error 111)
I20260810 22:36:36.004045 32424 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.233.124" port: 46485 } attempt: 2
I20260810 22:36:36.005237 32428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:36.004772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.124:46485 (local address 127.0.0.1:34026)
0810 22:36:36.005075 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:36.005146 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.124:46485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183}
I20260810 22:36:36.006165 32428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:36.005907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.124:46485 (local address 127.0.0.1:34030)
0810 22:36:36.006057 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:36.006108 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.124:46485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260810 22:36:36.006381 32424 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.233.124:46485: Network error: Client connection negotiation failed: client connection to 127.16.233.124:46485: connect: Connection refused (error 111)
I20260810 22:36:36.031678 32511 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.233.124" port: 46485 } attempt: 2
I20260810 22:36:36.032655 32515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:36.032315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.124:46485 (local address 127.0.0.1:34046)
0810 22:36:36.032495 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:36.032573 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.124:46485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260810 22:36:36.033607 32515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:36.033339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.124:46485 (local address 127.0.0.1:34056)
0810 22:36:36.033479 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:36.033540 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.124:46485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260810 22:36:36.033802 32511 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.233.124:46485: Network error: Client connection negotiation failed: client connection to 127.16.233.124:46485: connect: Connection refused (error 111)
W20260810 22:36:36.041785 32518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:36:36.042117 32518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:36:36.042196 32518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:36:36.046720 32518 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:36:36.046814 32518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:36:36.046835 32518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:36:36.046854 32518 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:36:36.046869 32518 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:36:36.052968 32518 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:36035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.126:44231,127.16.233.125:36301,127.16.233.124:46485
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.124.keytab
--principal=kudu/127.16.233.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.124:46485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.233.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:36:36.054867 32518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:36:36.056058 32518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 22:36:36.065919 32518 server_base.cc:1034] running on GCE node
W20260810 22:36:36.065732 32524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:36:36.065740 32523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:36:36.065788 32526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:36:36.066957 32518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:36:36.068370 32518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:36:36.069578 32518 hybrid_clock.cc:648] HybridClock initialized: now 1786401396069555 us; error 42 us; skew 500 ppm
Aug 10 22:36:36 dist-test-slave-77v9 krb5kdc[32332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401396, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:36:36.072914 32518 init.cc:377] Logged in from keytab as kudu/127.16.233.124@KRBTEST.COM (short username kudu)
I20260810 22:36:36.074050 32518 webserver.cc:460] Webserver started at http://127.16.233.124:46851/ using document root <none> and password file <none>
I20260810 22:36:36.074754 32518 fs_manager.cc:359] Metadata directory not provided
I20260810 22:36:36.074862 32518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:36:36.075124 32518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:36:36.077589 32518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/minicluster-data/master-2/data/instance:
uuid: "fba232acb8524ed4abe8353cc3168480"
format_stamp: "Formatted at 2026-08-10 22:36:36 on dist-test-slave-77v9"
server_key: "dac1e596d045cf2a0cd987117f74837e"
server_key_iv: "832cb11281e31d1ad2beac55ccf79b4a"
server_key_version: "encryptionkey@0"
I20260810 22:36:36.078274 32518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/minicluster-data/master-2/wal/instance:
uuid: "fba232acb8524ed4abe8353cc3168480"
format_stamp: "Formatted at 2026-08-10 22:36:36 on dist-test-slave-77v9"
server_key: "dac1e596d045cf2a0cd987117f74837e"
server_key_iv: "832cb11281e31d1ad2beac55ccf79b4a"
server_key_version: "encryptionkey@0"
I20260810 22:36:36.082758 32518 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 22:36:36.085613 32536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:36:36.087026 32518 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:36:36.087229 32518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/minicluster-data/master-2/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/minicluster-data/master-2/wal
uuid: "fba232acb8524ed4abe8353cc3168480"
format_stamp: "Formatted at 2026-08-10 22:36:36 on dist-test-slave-77v9"
server_key: "dac1e596d045cf2a0cd987117f74837e"
server_key_iv: "832cb11281e31d1ad2beac55ccf79b4a"
server_key_version: "encryptionkey@0"
I20260810 22:36:36.087378 32518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:36:36.105634 32424 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.233.124" port: 46485 } attempt: 3
I20260810 22:36:36.106786 32428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:36.106345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.124:46485 (local address 127.0.0.1:34060)
0810 22:36:36.106570 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:36.106651 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.124:46485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260810 22:36:36.107614 32428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:36.107357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.124:46485 (local address 127.0.0.1:34066)
0810 22:36:36.107494 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:36.107548 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.124:46485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260810 22:36:36.107801 32424 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.233.124:46485: Network error: Client connection negotiation failed: client connection to 127.16.233.124:46485: connect: Connection refused (error 111)
I20260810 22:36:36.110051 32518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:36:36.113703 32518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:36:36.113884 32518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:36:36.121805 32518 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.124:46485
I20260810 22:36:36.122192 32596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.124:46485 every 8 connection(s)
I20260810 22:36:36.122989 32518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/minicluster-data/master-2/data/info.pb
I20260810 22:36:36.124966 32511 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.233.124" port: 46485 } attempt: 3
I20260810 22:36:36.125767 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 32518
I20260810 22:36:36.125914 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786401215402979-17317-0/minicluster-data/master-2/wal/instance
I20260810 22:36:36.126236 17317 external_mini_cluster.cc:1428] Setting key f0ebcfbcfa6fe50026f3ad3b555ea954
I20260810 22:36:36.126473 32597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:36:36.131637 32597 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.233.126" port: 44231 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 22:36:36 dist-test-slave-77v9 krb5kdc[32332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401395, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.125@KRBTEST.COM for kudu/127.16.233.124@KRBTEST.COM
Aug 10 22:36:36 dist-test-slave-77v9 krb5kdc[32332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401395, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.124@KRBTEST.COM
Aug 10 22:36:36 dist-test-slave-77v9 krb5kdc[32332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401396, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.124@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:36:36.140651 32515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:36.125648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.124:46485 (local address 127.0.0.1:34080)
0810 22:36:36.125915 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:36.125933 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 22:36:36.126044 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:36:36.128935 (+ 2891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:36:36.128939 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:36:36.129167 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:36:36.129294 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:36.129318 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:36.130981 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:36.130984 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:36.131841 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:36.131853 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:36.132763 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:36.132767 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:36.132883 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:36.133552 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:36:36.133572 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:36:36.135289 (+ 1717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:36:36.138243 (+ 2954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:36:36.138247 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:36:36.138250 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:36:36.138541 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:36:36.139011 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:36:36.139014 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:36:36.139016 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:36:36.139112 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:36:36.140016 (+ 904us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:36:36.140022 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:36:36.140151 (+ 129us) client_negotiation.cc:769] Sending connection context
0810 22:36:36.140276 (+ 125us) client_negotiation.cc:240] Negotiation successful
0810 22:36:36.140444 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162}
I20260810 22:36:36.141988 32600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:36.126749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34080 (local address 127.16.233.124:46485)
0810 22:36:36.127431 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0810 22:36:36.127451 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 22:36:36.127494 (+ 43us) server_negotiation.cc:407] Connection header received
0810 22:36:36.128234 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:36:36.128239 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:36:36.128581 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:36:36.129437 (+ 856us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:36:36.129584 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:36.130800 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:36.131987 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:36.132601 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:36.133151 (+ 550us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:36.135525 (+ 2374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:36:36.135533 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:36:36.135545 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:36:36.135579 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:36:36.138048 (+ 2469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:36:36.138707 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:36:36.138715 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:36:36.138726 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:36:36.138889 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:36:36.139214 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:36:36.139219 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:36:36.139221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:36:36.139880 (+ 659us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:36:36.140024 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 22:36:36.140419 (+ 395us) server_negotiation.cc:299] Negotiation successful
0810 22:36:36.140665 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":125,"server-negotiator.queue_time_us":502,"spinlock_wait_cycles":68608,"thread_start_us":171,"threads_started":1}
I20260810 22:36:36.141988 32601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:36.126888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34082 (local address 127.16.233.124:46485)
0810 22:36:36.127430 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0810 22:36:36.127451 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 22:36:36.127496 (+ 45us) server_negotiation.cc:407] Connection header received
0810 22:36:36.128201 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:36:36.128218 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:36:36.128581 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:36:36.129000 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:36:36.129798 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:36.130773 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:36.132025 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:36.132599 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:36.132788 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:36.136450 (+ 3662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:36:36.136453 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:36:36.136456 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:36:36.136482 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:36:36.138300 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:36:36.138874 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:36:36.138878 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:36:36.138879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:36:36.138991 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:36:36.139344 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:36:36.139347 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:36:36.139349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:36:36.139880 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:36:36.140087 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0810 22:36:36.141209 (+ 1122us) server_negotiation.cc:299] Negotiation successful
0810 22:36:36.141331 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":446,"spinlock_wait_cycles":9216,"thread_start_us":157,"threads_started":1}
I20260810 22:36:36.145704 32602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:36.133806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:44231 (local address 127.0.0.1:33676)
0810 22:36:36.134454 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:36.134510 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 22:36:36.134688 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:36:36.135093 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:36:36.135099 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:36:36.135610 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:36:36.135737 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:36.135744 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:36.136685 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:36.136691 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:36.137690 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:36.137700 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:36.138208 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:36.138212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:36.138381 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:36.138925 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:36:36.138942 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:36:36.141315 (+ 2373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:36:36.143659 (+ 2344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:36:36.143668 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:36:36.143673 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:36:36.143974 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:36:36.144338 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:36:36.144341 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:36:36.144343 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:36:36.144445 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:36:36.145042 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:36:36.145048 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:36:36.145169 (+ 121us) client_negotiation.cc:769] Sending connection context
0810 22:36:36.145323 (+ 154us) client_negotiation.cc:240] Negotiation successful
0810 22:36:36.145526 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":129,"threads_started":1}
I20260810 22:36:36.145727 32429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:36.133921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33676 (local address 127.16.233.126:44231)
0810 22:36:36.134072 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 22:36:36.134078 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:36:36.134592 (+ 514us) server_negotiation.cc:407] Connection header received
0810 22:36:36.134793 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:36:36.134797 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:36:36.134854 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:36:36.134950 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:36:36.135941 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:36.136554 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:36.137825 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:36.138082 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:36.138205 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:36.141498 (+ 3293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:36:36.141502 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:36:36.141504 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:36:36.141536 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:36:36.143534 (+ 1998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:36:36.144093 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:36:36.144100 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:36:36.144104 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:36:36.144201 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:36:36.144577 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:36:36.144584 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:36:36.144589 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:36:36.144878 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:36:36.144957 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 22:36:36.145388 (+ 431us) server_negotiation.cc:299] Negotiation successful
0810 22:36:36.145547 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 22:36:36.146925 32511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60255f340e9a41049e72acdde7dff5cb: Bootstrap starting.
I20260810 22:36:36.147393 32597 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.233.125" port: 36301 } has no permanent_uuid. Determining permanent_uuid...
I20260810 22:36:36.150138 32511 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60255f340e9a41049e72acdde7dff5cb: Neither blocks nor log segments found. Creating new log.
I20260810 22:36:36.151166 32511 log.cc:824] T 00000000000000000000000000000000 P 60255f340e9a41049e72acdde7dff5cb: Log is configured to *not* fsync() on all Append() calls
Aug 10 22:36:36 dist-test-slave-77v9 krb5kdc[32332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401395, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
Aug 10 22:36:36 dist-test-slave-77v9 krb5kdc[32332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401396, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.124@KRBTEST.COM for kudu/127.16.233.125@KRBTEST.COM
I20260810 22:36:36.153874 32511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60255f340e9a41049e72acdde7dff5cb: No bootstrap required, opened a new log
I20260810 22:36:36.156484 32603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:36.144934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33678 (local address 127.16.233.126:44231)
0810 22:36:36.145257 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0810 22:36:36.145269 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 22:36:36.145307 (+ 38us) server_negotiation.cc:407] Connection header received
0810 22:36:36.145346 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:36:36.145348 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:36:36.145388 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:36:36.146228 (+ 840us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:36:36.146359 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:36.147183 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:36.148465 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:36.148845 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:36.149340 (+ 495us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:36.152164 (+ 2824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:36:36.152169 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:36:36.152172 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:36:36.152201 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:36:36.154338 (+ 2137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:36:36.154879 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:36:36.154882 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:36:36.154883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:36:36.154996 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:36:36.155313 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:36:36.155319 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:36:36.155324 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:36:36.155572 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:36:36.155793 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0810 22:36:36.155989 (+ 196us) server_negotiation.cc:299] Negotiation successful
0810 22:36:36.156130 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":109,"threads_started":1}
I20260810 22:36:36.156774 32602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:36.147935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.125:36301 (local address 127.0.0.1:39088)
0810 22:36:36.148116 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:36.148131 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:36:36.148216 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:36:36.148490 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:36:36.148493 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:36:36.148673 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:36:36.148767 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:36.148772 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:36.149647 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:36.149650 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:36.150192 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:36.150199 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:36.150799 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:36.150805 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:36.150903 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:36.151387 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:36:36.151407 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:36:36.152879 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:36:36.155011 (+ 2132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:36:36.155016 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:36:36.155020 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:36:36.155250 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:36:36.155552 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:36:36.155555 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:36:36.155557 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:36:36.155656 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:36:36.156221 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:36:36.156224 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:36:36.156345 (+ 121us) client_negotiation.cc:769] Sending connection context
0810 22:36:36.156447 (+ 102us) client_negotiation.cc:240] Negotiation successful
0810 22:36:36.156618 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113}
I20260810 22:36:36.156889 32513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:36.147994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39088 (local address 127.16.233.125:36301)
0810 22:36:36.148123 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 22:36:36.148129 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:36:36.148175 (+ 46us) server_negotiation.cc:407] Connection header received
0810 22:36:36.148337 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:36:36.148340 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:36:36.148383 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:36:36.148475 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:36:36.148934 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:36.149518 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:36.150347 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:36.150570 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:36.150626 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:36.153007 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:36:36.153013 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:36:36.153019 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:36:36.153051 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:36:36.154888 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:36:36.155367 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:36:36.155370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:36:36.155372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:36:36.155455 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:36:36.155771 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:36:36.155775 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:36:36.155776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:36:36.156062 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:36:36.156139 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 22:36:36.156557 (+ 418us) server_negotiation.cc:299] Negotiation successful
0810 22:36:36.156684 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 22:36:36.157644 32597 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.233.124" port: 46485 } has no permanent_uuid. Determining permanent_uuid...
I20260810 22:36:36.158577 32511 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60255f340e9a41049e72acdde7dff5cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea876dec6927461fb6e989b21abf5331" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 44231 } } peers { permanent_uuid: "60255f340e9a41049e72acdde7dff5cb" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 36301 } } peers { permanent_uuid: "fba232acb8524ed4abe8353cc3168480" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 46485 } }
I20260810 22:36:36.158878 32511 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60255f340e9a41049e72acdde7dff5cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:36:36.158952 32511 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60255f340e9a41049e72acdde7dff5cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60255f340e9a41049e72acdde7dff5cb, State: Initialized, Role: FOLLOWER
I20260810 22:36:36.159559 32511 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60255f340e9a41049e72acdde7dff5cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea876dec6927461fb6e989b21abf5331" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 44231 } } peers { permanent_uuid: "60255f340e9a41049e72acdde7dff5cb" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 36301 } } peers { permanent_uuid: "fba232acb8524ed4abe8353cc3168480" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 46485 } }
I20260810 22:36:36.160248 32605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60255f340e9a41049e72acdde7dff5cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea876dec6927461fb6e989b21abf5331" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 44231 } } peers { permanent_uuid: "60255f340e9a41049e72acdde7dff5cb" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 36301 } } peers { permanent_uuid: "fba232acb8524ed4abe8353cc3168480" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 46485 } } }
I20260810 22:36:36.160461 32605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60255f340e9a41049e72acdde7dff5cb [sys.catalog]: This master's current role is: FOLLOWER
I20260810 22:36:36.161213 32511 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60255f340e9a41049e72acdde7dff5cb [sys.catalog]: configured and running, proceeding with master startup.
Aug 10 22:36:36 dist-test-slave-77v9 krb5kdc[32332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401396, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.124@KRBTEST.COM for kudu/127.16.233.124@KRBTEST.COM
Aug 10 22:36:36 dist-test-slave-77v9 krb5kdc[32332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401395, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.125@KRBTEST.COM
W20260810 22:36:36.167866 32617 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 60255f340e9a41049e72acdde7dff5cb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 22:36:36.168026 32617 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 22:36:36.165436 32435 master_runner.cc:186] Error getting master registration for 127.16.233.126:44231: OK, instance_id { permanent_uuid: "ea876dec6927461fb6e989b21abf5331" instance_seqno: 1786401395620853 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 22:36:36.168918 32513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:36.158543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39104 (local address 127.16.233.125:36301)
0810 22:36:36.158650 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0810 22:36:36.158654 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:36:36.158668 (+ 14us) server_negotiation.cc:407] Connection header received
0810 22:36:36.158760 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:36:36.158763 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:36:36.158815 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:36:36.158898 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:36:36.159838 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:36.160529 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:36.161657 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:36.161927 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:36.162001 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:36.165059 (+ 3058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:36:36.165063 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:36:36.165066 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:36:36.165093 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:36:36.166747 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:36:36.167368 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:36:36.167372 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:36:36.167374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:36:36.167463 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:36:36.167791 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:36:36.167794 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:36:36.167796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:36:36.168033 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:36:36.168177 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 22:36:36.168588 (+ 411us) server_negotiation.cc:299] Negotiation successful
0810 22:36:36.168729 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 22:36:36.169001 32435 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 22:36:36.171900 32602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:36.158164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.124:46485 (local address 127.0.0.1:34094)
0810 22:36:36.158312 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:36.158328 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 22:36:36.158407 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:36:36.158819 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:36:36.158822 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:36:36.159006 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:36:36.159097 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:36.159103 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:36.159895 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:36.159898 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:36.160676 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:36.160688 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:36.161423 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:36.161426 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:36.161494 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:36.162060 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:36:36.162078 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:36:36.163724 (+ 1646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:36:36.169816 (+ 6092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:36:36.169821 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:36:36.169823 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:36:36.170069 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:36:36.170410 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:36:36.170413 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:36:36.170415 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:36:36.170497 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:36:36.171263 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:36:36.171265 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:36:36.171393 (+ 128us) client_negotiation.cc:769] Sending connection context
0810 22:36:36.171514 (+ 121us) client_negotiation.cc:240] Negotiation successful
0810 22:36:36.171694 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
I20260810 22:36:36.171950 32601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:36.158187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34094 (local address 127.16.233.124:46485)
0810 22:36:36.158627 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0810 22:36:36.158634 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 22:36:36.158648 (+ 14us) server_negotiation.cc:407] Connection header received
0810 22:36:36.158680 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:36:36.158683 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:36:36.158729 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:36:36.158813 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:36:36.159209 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:36.159760 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:36.160876 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:36.161225 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:36.161322 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:36.163900 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:36:36.163905 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:36:36.163909 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:36:36.163939 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:36:36.169703 (+ 5764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:36:36.170183 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:36:36.170188 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:36:36.170190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:36:36.170300 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:36:36.170588 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:36:36.170590 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:36:36.170592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:36:36.170815 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:36:36.170932 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 22:36:36.171598 (+ 666us) server_negotiation.cc:299] Negotiation successful
0810 22:36:36.171774 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":33}
I20260810 22:36:36.173564 17317 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 22:36:36.175545 32597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fba232acb8524ed4abe8353cc3168480: Bootstrap starting.
I20260810 22:36:36.178843 32597 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fba232acb8524ed4abe8353cc3168480: Neither blocks nor log segments found. Creating new log.
I20260810 22:36:36.180099 32597 log.cc:824] T 00000000000000000000000000000000 P fba232acb8524ed4abe8353cc3168480: Log is configured to *not* fsync() on all Append() calls
I20260810 22:36:36.182665 32597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fba232acb8524ed4abe8353cc3168480: No bootstrap required, opened a new log
I20260810 22:36:36.187007 32597 raft_consensus.cc:348] T 00000000000000000000000000000000 P fba232acb8524ed4abe8353cc3168480 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea876dec6927461fb6e989b21abf5331" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 44231 } } peers { permanent_uuid: "60255f340e9a41049e72acdde7dff5cb" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 36301 } } peers { permanent_uuid: "fba232acb8524ed4abe8353cc3168480" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 46485 } }
I20260810 22:36:36.187271 32597 raft_consensus.cc:374] T 00000000000000000000000000000000 P fba232acb8524ed4abe8353cc3168480 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:36:36.187330 32597 raft_consensus.cc:729] T 00000000000000000000000000000000 P fba232acb8524ed4abe8353cc3168480 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fba232acb8524ed4abe8353cc3168480, State: Initialized, Role: FOLLOWER
I20260810 22:36:36.187631 32603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:36.176096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33680 (local address 127.16.233.126:44231)
0810 22:36:36.176326 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0810 22:36:36.176329 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:36:36.177754 (+ 1425us) server_negotiation.cc:407] Connection header received
0810 22:36:36.177942 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:36:36.177945 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:36:36.178000 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:36:36.178052 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:36:36.178990 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:36.179870 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:36.181027 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:36.181444 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:36.181535 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:36.183383 (+ 1848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:36:36.183388 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:36:36.183392 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:36:36.183425 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:36:36.185300 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:36:36.185943 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:36:36.185947 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:36:36.185949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:36:36.186048 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:36:36.186405 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:36:36.186409 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:36:36.186411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:36:36.186663 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:36:36.186798 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0810 22:36:36.187331 (+ 533us) server_negotiation.cc:299] Negotiation successful
0810 22:36:36.187461 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134}
I20260810 22:36:36.187630 32601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:36.176431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34108 (local address 127.16.233.124:46485)
0810 22:36:36.176554 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0810 22:36:36.176558 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:36:36.176572 (+ 14us) server_negotiation.cc:407] Connection header received
0810 22:36:36.176613 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:36:36.176618 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:36:36.176672 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:36:36.176754 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:36:36.177903 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:36.178708 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:36.180013 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:36.180346 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:36.180428 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:36.182917 (+ 2489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:36:36.182926 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:36:36.182932 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:36:36.182966 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:36:36.185300 (+ 2334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:36:36.185960 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:36:36.185964 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:36:36.185965 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:36:36.186048 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:36:36.186405 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:36:36.186409 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:36:36.186411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:36:36.186663 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:36:36.186772 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 22:36:36.187324 (+ 552us) server_negotiation.cc:299] Negotiation successful
0810 22:36:36.187461 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 22:36:36.188421 32597 consensus_queue.cc:260] T 00000000000000000000000000000000 P fba232acb8524ed4abe8353cc3168480 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea876dec6927461fb6e989b21abf5331" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 44231 } } peers { permanent_uuid: "60255f340e9a41049e72acdde7dff5cb" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 36301 } } peers { permanent_uuid: "fba232acb8524ed4abe8353cc3168480" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 46485 } }
I20260810 22:36:36.190058 32597 sys_catalog.cc:564] T 00000000000000000000000000000000 P fba232acb8524ed4abe8353cc3168480 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:36:36.193754 32629 sys_catalog.cc:455] T 00000000000000000000000000000000 P fba232acb8524ed4abe8353cc3168480 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea876dec6927461fb6e989b21abf5331" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 44231 } } peers { permanent_uuid: "60255f340e9a41049e72acdde7dff5cb" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 36301 } } peers { permanent_uuid: "fba232acb8524ed4abe8353cc3168480" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 46485 } } }
I20260810 22:36:36.194002 32629 sys_catalog.cc:458] T 00000000000000000000000000000000 P fba232acb8524ed4abe8353cc3168480 [sys.catalog]: This master's current role is: FOLLOWER
W20260810 22:36:36.196255 32642 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fba232acb8524ed4abe8353cc3168480: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 22:36:36.196412 32642 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 22:36:36.194072 32518 master_runner.cc:186] Error getting master registration for 127.16.233.126:44231: OK, instance_id { permanent_uuid: "ea876dec6927461fb6e989b21abf5331" instance_seqno: 1786401395620853 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 22:36:36.197616 32518 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 22:36:36.200176 32513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:36.177071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39106 (local address 127.16.233.125:36301)
0810 22:36:36.177208 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 22:36:36.177214 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:36:36.188204 (+ 10990us) server_negotiation.cc:407] Connection header received
0810 22:36:36.188414 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:36:36.188420 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:36:36.188480 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:36:36.188535 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:36:36.189525 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:36.190231 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:36.191262 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:36.191544 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:36.191609 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:36.193565 (+ 1956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:36:36.193569 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:36:36.193572 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:36:36.193603 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:36:36.198205 (+ 4602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:36:36.198767 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:36:36.198771 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:36:36.198773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:36:36.198875 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:36:36.199202 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:36:36.199206 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:36:36.199208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:36:36.199429 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:36:36.199598 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0810 22:36:36.199870 (+ 272us) server_negotiation.cc:299] Negotiation successful
0810 22:36:36.200003 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 22:36:36.271989 32424 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.233.124" port: 46485 } attempt: 4
Aug 10 22:36:36 dist-test-slave-77v9 krb5kdc[32332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401395, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.126@KRBTEST.COM for kudu/127.16.233.124@KRBTEST.COM
I20260810 22:36:36.282722 32428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:36.272641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.124:46485 (local address 127.0.0.1:34114)
0810 22:36:36.272962 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:36.272987 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 22:36:36.273129 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:36:36.273435 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:36:36.273439 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:36:36.273675 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:36:36.273776 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:36.273782 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:36.274776 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:36.274781 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:36.275587 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:36.275599 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:36.276142 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:36.276146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:36.276241 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:36.276907 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:36:36.276922 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:36:36.278392 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:36:36.280577 (+ 2185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:36:36.280581 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:36:36.280583 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:36:36.280875 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:36:36.281422 (+ 547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:36:36.281426 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:36:36.281428 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:36:36.281540 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:36:36.282100 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:36:36.282103 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:36:36.282225 (+ 122us) client_negotiation.cc:769] Sending connection context
0810 22:36:36.282329 (+ 104us) client_negotiation.cc:240] Negotiation successful
0810 22:36:36.282521 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119}
I20260810 22:36:36.284269 32601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:36.272704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34114 (local address 127.16.233.124:46485)
0810 22:36:36.272908 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0810 22:36:36.272913 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:36:36.273064 (+ 151us) server_negotiation.cc:407] Connection header received
0810 22:36:36.273263 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:36:36.273266 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:36:36.273312 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:36:36.273417 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:36:36.273947 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:36.274650 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:36.275727 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:36.275997 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:36.276122 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:36.278573 (+ 2451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:36:36.278577 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:36:36.278579 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:36:36.278607 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:36:36.280433 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:36:36.281160 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:36:36.281165 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:36:36.281167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:36:36.281278 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:36:36.281671 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:36:36.281674 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:36:36.281676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:36:36.281942 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:36:36.282045 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 22:36:36.283952 (+ 1907us) server_negotiation.cc:299] Negotiation successful
0810 22:36:36.284115 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260810 22:36:36.287552 32424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea876dec6927461fb6e989b21abf5331: Bootstrap starting.
I20260810 22:36:36.290630 32424 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea876dec6927461fb6e989b21abf5331: Neither blocks nor log segments found. Creating new log.
I20260810 22:36:36.291854 32424 log.cc:824] T 00000000000000000000000000000000 P ea876dec6927461fb6e989b21abf5331: Log is configured to *not* fsync() on all Append() calls
I20260810 22:36:36.294387 32424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea876dec6927461fb6e989b21abf5331: No bootstrap required, opened a new log
I20260810 22:36:36.299638 32424 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea876dec6927461fb6e989b21abf5331 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea876dec6927461fb6e989b21abf5331" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 44231 } } peers { permanent_uuid: "60255f340e9a41049e72acdde7dff5cb" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 36301 } } peers { permanent_uuid: "fba232acb8524ed4abe8353cc3168480" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 46485 } }
I20260810 22:36:36.299944 32424 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea876dec6927461fb6e989b21abf5331 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:36:36.300062 32424 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea876dec6927461fb6e989b21abf5331 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea876dec6927461fb6e989b21abf5331, State: Initialized, Role: FOLLOWER
I20260810 22:36:36.300630 32424 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea876dec6927461fb6e989b21abf5331 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea876dec6927461fb6e989b21abf5331" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 44231 } } peers { permanent_uuid: "60255f340e9a41049e72acdde7dff5cb" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 36301 } } peers { permanent_uuid: "fba232acb8524ed4abe8353cc3168480" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 46485 } }
I20260810 22:36:36.301268 32645 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea876dec6927461fb6e989b21abf5331 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea876dec6927461fb6e989b21abf5331" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 44231 } } peers { permanent_uuid: "60255f340e9a41049e72acdde7dff5cb" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 36301 } } peers { permanent_uuid: "fba232acb8524ed4abe8353cc3168480" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 46485 } } }
I20260810 22:36:36.301543 32645 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea876dec6927461fb6e989b21abf5331 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 22:36:36.302326 32424 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea876dec6927461fb6e989b21abf5331 [sys.catalog]: configured and running, proceeding with master startup.
W20260810 22:36:36.308434 32658 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ea876dec6927461fb6e989b21abf5331: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 22:36:36.308645 32658 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 22:36:36.312916 32350 master_runner.cc:263] No leader master found from master ea876dec6927461fb6e989b21abf5331
I20260810 22:36:36.313028 32350 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260810 22:36:36.365520 32543 tablet.cc:2369] T 00000000000000000000000000000000 P fba232acb8524ed4abe8353cc3168480: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 22:36:36.366314 32458 tablet.cc:2369] T 00000000000000000000000000000000 P 60255f340e9a41049e72acdde7dff5cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 22:36:36.366976 32373 tablet.cc:2369] T 00000000000000000000000000000000 P ea876dec6927461fb6e989b21abf5331: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:36:36.488931 32645 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea876dec6927461fb6e989b21abf5331 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:36:36.489228 32645 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea876dec6927461fb6e989b21abf5331 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea876dec6927461fb6e989b21abf5331" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 44231 } } peers { permanent_uuid: "60255f340e9a41049e72acdde7dff5cb" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 36301 } } peers { permanent_uuid: "fba232acb8524ed4abe8353cc3168480" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 46485 } }
I20260810 22:36:36.490432 32645 leader_election.cc:288] T 00000000000000000000000000000000 P ea876dec6927461fb6e989b21abf5331 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60255f340e9a41049e72acdde7dff5cb (127.16.233.125:36301), fba232acb8524ed4abe8353cc3168480 (127.16.233.124:46485)
I20260810 22:36:36.491097 32481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ea876dec6927461fb6e989b21abf5331" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60255f340e9a41049e72acdde7dff5cb" is_pre_election: true
I20260810 22:36:36.491096 32567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ea876dec6927461fb6e989b21abf5331" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fba232acb8524ed4abe8353cc3168480" is_pre_election: true
I20260810 22:36:36.491461 32481 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 60255f340e9a41049e72acdde7dff5cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea876dec6927461fb6e989b21abf5331 in term 0.
I20260810 22:36:36.491503 32567 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fba232acb8524ed4abe8353cc3168480 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea876dec6927461fb6e989b21abf5331 in term 0.
I20260810 22:36:36.492079 32367 leader_election.cc:302] T 00000000000000000000000000000000 P ea876dec6927461fb6e989b21abf5331 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ea876dec6927461fb6e989b21abf5331, fba232acb8524ed4abe8353cc3168480; no voters:
I20260810 22:36:36.492443 32645 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea876dec6927461fb6e989b21abf5331 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:36:36.492542 32645 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea876dec6927461fb6e989b21abf5331 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:36:36.492586 32645 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea876dec6927461fb6e989b21abf5331 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:36:36.493744 32645 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea876dec6927461fb6e989b21abf5331 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea876dec6927461fb6e989b21abf5331" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 44231 } } peers { permanent_uuid: "60255f340e9a41049e72acdde7dff5cb" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 36301 } } peers { permanent_uuid: "fba232acb8524ed4abe8353cc3168480" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 46485 } }
I20260810 22:36:36.494141 32645 leader_election.cc:288] T 00000000000000000000000000000000 P ea876dec6927461fb6e989b21abf5331 [CANDIDATE]: Term 1 election: Requested vote from peers 60255f340e9a41049e72acdde7dff5cb (127.16.233.125:36301), fba232acb8524ed4abe8353cc3168480 (127.16.233.124:46485)
I20260810 22:36:36.494567 32567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ea876dec6927461fb6e989b21abf5331" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fba232acb8524ed4abe8353cc3168480"
I20260810 22:36:36.494576 32481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ea876dec6927461fb6e989b21abf5331" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60255f340e9a41049e72acdde7dff5cb"
I20260810 22:36:36.494705 32567 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fba232acb8524ed4abe8353cc3168480 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:36:36.494755 32481 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60255f340e9a41049e72acdde7dff5cb [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:36:36.495947 32567 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fba232acb8524ed4abe8353cc3168480 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea876dec6927461fb6e989b21abf5331 in term 1.
I20260810 22:36:36.496006 32481 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 60255f340e9a41049e72acdde7dff5cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea876dec6927461fb6e989b21abf5331 in term 1.
I20260810 22:36:36.496404 32367 leader_election.cc:302] T 00000000000000000000000000000000 P ea876dec6927461fb6e989b21abf5331 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ea876dec6927461fb6e989b21abf5331, fba232acb8524ed4abe8353cc3168480; no voters:
I20260810 22:36:36.496649 32645 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea876dec6927461fb6e989b21abf5331 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:36:36.496974 32645 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea876dec6927461fb6e989b21abf5331 [term 1 LEADER]: Becoming Leader. State: Replica: ea876dec6927461fb6e989b21abf5331, State: Running, Role: LEADER
I20260810 22:36:36.497375 32645 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea876dec6927461fb6e989b21abf5331 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea876dec6927461fb6e989b21abf5331" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 44231 } } peers { permanent_uuid: "60255f340e9a41049e72acdde7dff5cb" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 36301 } } peers { permanent_uuid: "fba232acb8524ed4abe8353cc3168480" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 46485 } }
I20260810 22:36:36.499516 32661 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea876dec6927461fb6e989b21abf5331 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea876dec6927461fb6e989b21abf5331. Latest consensus state: current_term: 1 leader_uuid: "ea876dec6927461fb6e989b21abf5331" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea876dec6927461fb6e989b21abf5331" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 44231 } } peers { permanent_uuid: "60255f340e9a41049e72acdde7dff5cb" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 36301 } } peers { permanent_uuid: "fba232acb8524ed4abe8353cc3168480" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 46485 } } }
I20260810 22:36:36.499672 32661 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea876dec6927461fb6e989b21abf5331 [sys.catalog]: This master's current role is: LEADER
I20260810 22:36:36.500111 32663 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:36:36.501093 32663 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:36:36.506510 32567 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fba232acb8524ed4abe8353cc3168480 [term 1 FOLLOWER]: Refusing update from remote peer ea876dec6927461fb6e989b21abf5331: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:36:36.506616 32481 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 60255f340e9a41049e72acdde7dff5cb [term 1 FOLLOWER]: Refusing update from remote peer ea876dec6927461fb6e989b21abf5331: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:36:36.507288 32661 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ea876dec6927461fb6e989b21abf5331 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60255f340e9a41049e72acdde7dff5cb" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 36301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:36:36.507526 32645 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ea876dec6927461fb6e989b21abf5331 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fba232acb8524ed4abe8353cc3168480" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 46485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:36:36.511837 32629 sys_catalog.cc:455] T 00000000000000000000000000000000 P fba232acb8524ed4abe8353cc3168480 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea876dec6927461fb6e989b21abf5331. Latest consensus state: current_term: 1 leader_uuid: "ea876dec6927461fb6e989b21abf5331" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea876dec6927461fb6e989b21abf5331" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 44231 } } peers { permanent_uuid: "60255f340e9a41049e72acdde7dff5cb" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 36301 } } peers { permanent_uuid: "fba232acb8524ed4abe8353cc3168480" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 46485 } } }
I20260810 22:36:36.512015 32629 sys_catalog.cc:458] T 00000000000000000000000000000000 P fba232acb8524ed4abe8353cc3168480 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 22:36:36.512120 32605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60255f340e9a41049e72acdde7dff5cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea876dec6927461fb6e989b21abf5331. Latest consensus state: current_term: 1 leader_uuid: "ea876dec6927461fb6e989b21abf5331" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea876dec6927461fb6e989b21abf5331" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 44231 } } peers { permanent_uuid: "60255f340e9a41049e72acdde7dff5cb" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 36301 } } peers { permanent_uuid: "fba232acb8524ed4abe8353cc3168480" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 46485 } } }
I20260810 22:36:36.512283 32605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60255f340e9a41049e72acdde7dff5cb [sys.catalog]: This master's current role is: FOLLOWER
I20260810 22:36:36.515307 32670 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea876dec6927461fb6e989b21abf5331 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ea876dec6927461fb6e989b21abf5331" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea876dec6927461fb6e989b21abf5331" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 44231 } } peers { permanent_uuid: "60255f340e9a41049e72acdde7dff5cb" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 36301 } } peers { permanent_uuid: "fba232acb8524ed4abe8353cc3168480" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 46485 } } }
I20260810 22:36:36.515549 32670 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea876dec6927461fb6e989b21abf5331 [sys.catalog]: This master's current role is: LEADER
I20260810 22:36:36.515601 32645 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea876dec6927461fb6e989b21abf5331 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ea876dec6927461fb6e989b21abf5331" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea876dec6927461fb6e989b21abf5331" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 44231 } } peers { permanent_uuid: "60255f340e9a41049e72acdde7dff5cb" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 36301 } } peers { permanent_uuid: "fba232acb8524ed4abe8353cc3168480" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 46485 } } }
I20260810 22:36:36.515695 32645 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea876dec6927461fb6e989b21abf5331 [sys.catalog]: This master's current role is: LEADER
I20260810 22:36:36.516026 32629 sys_catalog.cc:455] T 00000000000000000000000000000000 P fba232acb8524ed4abe8353cc3168480 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ea876dec6927461fb6e989b21abf5331" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea876dec6927461fb6e989b21abf5331" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 44231 } } peers { permanent_uuid: "60255f340e9a41049e72acdde7dff5cb" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 36301 } } peers { permanent_uuid: "fba232acb8524ed4abe8353cc3168480" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 46485 } } }
I20260810 22:36:36.516160 32629 sys_catalog.cc:458] T 00000000000000000000000000000000 P fba232acb8524ed4abe8353cc3168480 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 22:36:36.516407 32666 mvcc.cc:205] Tried to move back new op lower bound from 7317100120083836928 to 7317100120056008704. Current Snapshot: MvccSnapshot[applied={T|T < 7317100120083836928}]
I20260810 22:36:36.516676 32605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60255f340e9a41049e72acdde7dff5cb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ea876dec6927461fb6e989b21abf5331" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea876dec6927461fb6e989b21abf5331" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 44231 } } peers { permanent_uuid: "60255f340e9a41049e72acdde7dff5cb" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 36301 } } peers { permanent_uuid: "fba232acb8524ed4abe8353cc3168480" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 46485 } } }
I20260810 22:36:36.516799 32663 catalog_manager.cc:1347] Generated new cluster ID: d8d3f67f61f84cff9fc622688a0a0a89
I20260810 22:36:36.516803 32605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60255f340e9a41049e72acdde7dff5cb [sys.catalog]: This master's current role is: FOLLOWER
I20260810 22:36:36.516889 32663 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:36:36.521044 32664 mvcc.cc:205] Tried to move back new op lower bound from 7317100120083836928 to 7317100120056008704. Current Snapshot: MvccSnapshot[applied={T|T < 7317100120083836928 or (T in {7317100120083836928})}]
I20260810 22:36:36.538388 32663 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:36:36.539511 32663 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:36:36.549962 32663 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea876dec6927461fb6e989b21abf5331: Generated new TSK 0
I20260810 22:36:36.551064 32663 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:36:37.169638 32617 catalog_manager.cc:1259] Loaded cluster ID: d8d3f67f61f84cff9fc622688a0a0a89
I20260810 22:36:37.169735 32617 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 60255f340e9a41049e72acdde7dff5cb: loading cluster ID for follower catalog manager: success
I20260810 22:36:37.171566 32617 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 60255f340e9a41049e72acdde7dff5cb: acquiring CA information for follower catalog manager: success
I20260810 22:36:37.172783 32617 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 60255f340e9a41049e72acdde7dff5cb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 22:36:37.197911 32642 catalog_manager.cc:1259] Loaded cluster ID: d8d3f67f61f84cff9fc622688a0a0a89
I20260810 22:36:37.198014 32642 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fba232acb8524ed4abe8353cc3168480: loading cluster ID for follower catalog manager: success
I20260810 22:36:37.199747 32642 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fba232acb8524ed4abe8353cc3168480: acquiring CA information for follower catalog manager: success
I20260810 22:36:37.201016 32642 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fba232acb8524ed4abe8353cc3168480: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 22:36:37.619189 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 32350
I20260810 22:36:37.629122 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 32435
I20260810 22:36:37.638594 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 32518
2026-08-10T22:36:37Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3558 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786401215402979-17317-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:36:37 dist-test-slave-77v9 krb5kdc[32682](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:36:37 dist-test-slave-77v9 krb5kdc[32682](info): set up 2 sockets
Aug 10 22:36:37 dist-test-slave-77v9 krb5kdc[32682](info): commencing operation
krb5kdc: starting...
W20260810 22:36:38.801793 17317 mini_kdc.cc:121] Time spent starting KDC: real 1.137s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Aug 10 22:36:38 dist-test-slave-77v9 krb5kdc[32682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401398, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:36:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:36:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.233.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab.
Entry for principal kudu/127.16.233.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab.
I20260810 22:36:38.949361 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786401215402979-17317-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786401215402979-17317-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786401215402979-17317-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:38943
--webserver_interface=127.16.233.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab
--principal=kudu/127.16.233.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:34399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.126:38943
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:36:39.072427 32697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:36:39.072978 32697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:36:39.073071 32697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:36:39.077935 32697 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:36:39.078049 32697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:36:39.078085 32697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:36:39.078128 32697 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:36:39.078163 32697 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:36:39.084118 32697 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:34399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786401215402979-17317-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786401215402979-17317-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.126:38943
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab
--principal=kudu/127.16.233.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:38943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786401215402979-17317-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:36:39.085537 32697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:36:39.086715 32697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:36:39.096714 32706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:36:39.096714 32703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:36:39.096736 32702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:36:39.096909 32697 server_base.cc:1034] running on GCE node
I20260810 22:36:39.098037 32697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:36:39.099378 32697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:36:39.100584 32697 hybrid_clock.cc:648] HybridClock initialized: now 1786401399100555 us; error 63 us; skew 500 ppm
Aug 10 22:36:39 dist-test-slave-77v9 krb5kdc[32682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401399, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:36:39.104200 32697 init.cc:377] Logged in from keytab as kudu/127.16.233.126@KRBTEST.COM (short username kudu)
I20260810 22:36:39.105346 32697 webserver.cc:460] Webserver started at http://127.16.233.126:36047/ using document root <none> and password file <none>
I20260810 22:36:39.105953 32697 fs_manager.cc:359] Metadata directory not provided
I20260810 22:36:39.106024 32697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:36:39.106225 32697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:36:39.108705 32697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786401215402979-17317-0/minicluster-data/master-0/data/instance:
uuid: "ad80a55fcbb1445a89c8cb28664e4a7a"
format_stamp: "Formatted at 2026-08-10 22:36:39 on dist-test-slave-77v9"
server_key: "35ee65c54f0b42bb34b392e53692591e"
server_key_iv: "cd205d208428b5710658ac914e0ebc66"
server_key_version: "encryptionkey@0"
I20260810 22:36:39.109437 32697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786401215402979-17317-0/minicluster-data/master-0/wal/instance:
uuid: "ad80a55fcbb1445a89c8cb28664e4a7a"
format_stamp: "Formatted at 2026-08-10 22:36:39 on dist-test-slave-77v9"
server_key: "35ee65c54f0b42bb34b392e53692591e"
server_key_iv: "cd205d208428b5710658ac914e0ebc66"
server_key_version: "encryptionkey@0"
I20260810 22:36:39.113955 32697 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 22:36:39.117148 32716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:36:39.118855 32697 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 22:36:39.119057 32697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786401215402979-17317-0/minicluster-data/master-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786401215402979-17317-0/minicluster-data/master-0/wal
uuid: "ad80a55fcbb1445a89c8cb28664e4a7a"
format_stamp: "Formatted at 2026-08-10 22:36:39 on dist-test-slave-77v9"
server_key: "35ee65c54f0b42bb34b392e53692591e"
server_key_iv: "cd205d208428b5710658ac914e0ebc66"
server_key_version: "encryptionkey@0"
I20260810 22:36:39.119216 32697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786401215402979-17317-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786401215402979-17317-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786401215402979-17317-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:36:39.151561 32697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:36:39.155894 32697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:36:39.156176 32697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:36:39.165438 32697 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.126:38943
I20260810 22:36:39.165457 305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.126:38943 every 8 connection(s)
I20260810 22:36:39.166669 32697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
I20260810 22:36:39.169008 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 32697
I20260810 22:36:39.169121 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786401215402979-17317-0/minicluster-data/master-0/wal/instance
I20260810 22:36:39.169452 17317 external_mini_cluster.cc:1428] Setting key 1fc44fef652168911e99b8cf1cb87334
I20260810 22:36:39.170181 306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:36:39.178640 306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad80a55fcbb1445a89c8cb28664e4a7a: Bootstrap starting.
Aug 10 22:36:39 dist-test-slave-77v9 krb5kdc[32682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401398, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:36:39.181903 306 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad80a55fcbb1445a89c8cb28664e4a7a: Neither blocks nor log segments found. Creating new log.
I20260810 22:36:39.183033 306 log.cc:824] T 00000000000000000000000000000000 P ad80a55fcbb1445a89c8cb28664e4a7a: Log is configured to *not* fsync() on all Append() calls
I20260810 22:36:39.186230 306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad80a55fcbb1445a89c8cb28664e4a7a: No bootstrap required, opened a new log
I20260810 22:36:39.188000 311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:39.171091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58066 (local address 127.16.233.126:38943)
0810 22:36:39.171606 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0810 22:36:39.171624 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 22:36:39.171664 (+ 40us) server_negotiation.cc:407] Connection header received
0810 22:36:39.172404 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:36:39.172425 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:36:39.172949 (+ 524us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:36:39.173292 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:36:39.174402 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:39.175484 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:39.176799 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:39.177589 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:39.177741 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:39.181374 (+ 3633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:36:39.181381 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:36:39.181397 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:36:39.181437 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:36:39.183872 (+ 2435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:36:39.184583 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:36:39.184591 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:36:39.184598 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:36:39.184771 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:36:39.185155 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:36:39.185161 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:36:39.185165 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:36:39.185751 (+ 586us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:36:39.185958 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0810 22:36:39.186367 (+ 409us) server_negotiation.cc:299] Negotiation successful
0810 22:36:39.186951 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":195,"threads_started":1}
I20260810 22:36:39.191387 306 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad80a55fcbb1445a89c8cb28664e4a7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad80a55fcbb1445a89c8cb28664e4a7a" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 38943 } }
I20260810 22:36:39.191622 17317 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 22:36:39.191764 306 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad80a55fcbb1445a89c8cb28664e4a7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:36:39.191874 306 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad80a55fcbb1445a89c8cb28664e4a7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad80a55fcbb1445a89c8cb28664e4a7a, State: Initialized, Role: FOLLOWER
I20260810 22:36:39.192333 306 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad80a55fcbb1445a89c8cb28664e4a7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad80a55fcbb1445a89c8cb28664e4a7a" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 38943 } }
I20260810 22:36:39.192504 306 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad80a55fcbb1445a89c8cb28664e4a7a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:36:39.192600 306 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad80a55fcbb1445a89c8cb28664e4a7a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:36:39.192745 306 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad80a55fcbb1445a89c8cb28664e4a7a [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:36:39.194104 306 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad80a55fcbb1445a89c8cb28664e4a7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad80a55fcbb1445a89c8cb28664e4a7a" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 38943 } }
I20260810 22:36:39.194486 306 leader_election.cc:302] T 00000000000000000000000000000000 P ad80a55fcbb1445a89c8cb28664e4a7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad80a55fcbb1445a89c8cb28664e4a7a; no voters:
I20260810 22:36:39.194862 306 leader_election.cc:288] T 00000000000000000000000000000000 P ad80a55fcbb1445a89c8cb28664e4a7a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:36:39.195022 313 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad80a55fcbb1445a89c8cb28664e4a7a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:36:39.195297 313 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad80a55fcbb1445a89c8cb28664e4a7a [term 1 LEADER]: Becoming Leader. State: Replica: ad80a55fcbb1445a89c8cb28664e4a7a, State: Running, Role: LEADER
I20260810 22:36:39.195662 313 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad80a55fcbb1445a89c8cb28664e4a7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad80a55fcbb1445a89c8cb28664e4a7a" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 38943 } }
I20260810 22:36:39.196230 306 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad80a55fcbb1445a89c8cb28664e4a7a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:36:39.197501 315 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad80a55fcbb1445a89c8cb28664e4a7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad80a55fcbb1445a89c8cb28664e4a7a. Latest consensus state: current_term: 1 leader_uuid: "ad80a55fcbb1445a89c8cb28664e4a7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad80a55fcbb1445a89c8cb28664e4a7a" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 38943 } } }
I20260810 22:36:39.197649 315 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad80a55fcbb1445a89c8cb28664e4a7a [sys.catalog]: This master's current role is: LEADER
I20260810 22:36:39.198074 324 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:36:39.198042 314 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad80a55fcbb1445a89c8cb28664e4a7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad80a55fcbb1445a89c8cb28664e4a7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad80a55fcbb1445a89c8cb28664e4a7a" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 38943 } } }
I20260810 22:36:39.198868 314 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad80a55fcbb1445a89c8cb28664e4a7a [sys.catalog]: This master's current role is: LEADER
I20260810 22:36:39.201123 324 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:36:39.207180 324 catalog_manager.cc:1347] Generated new cluster ID: 695df138dccd4805b5e48478d6d89525
I20260810 22:36:39.207314 324 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:36:39.220996 324 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:36:39.222105 324 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:36:39.229900 324 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad80a55fcbb1445a89c8cb28664e4a7a: Generated new TSK 0
I20260810 22:36:39.230756 324 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:36:39.248929 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 32697
2026-08-10T22:36:39Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1609 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:36:39 dist-test-slave-77v9 krb5kdc[342](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:36:39 dist-test-slave-77v9 krb5kdc[342](info): set up 2 sockets
Aug 10 22:36:39 dist-test-slave-77v9 krb5kdc[342](info): commencing operation
krb5kdc: starting...
W20260810 22:36:40.386121 17317 mini_kdc.cc:121] Time spent starting KDC: real 1.111s user 0.009s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Aug 10 22:36:40 dist-test-slave-77v9 krb5kdc[342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:36:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:36:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.233.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab.
Entry for principal kudu/127.16.233.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab.
I20260810 22:36:40.537421 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:42443
--webserver_interface=127.16.233.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab
--principal=kudu/127.16.233.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:33229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.126:42443
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:36:40.660246 359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:36:40.660581 359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:36:40.660626 359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:36:40.665074 359 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:36:40.665199 359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:36:40.665238 359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:36:40.665269 359 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:36:40.665298 359 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:36:40.671327 359 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:33229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.126:42443
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab
--principal=kudu/127.16.233.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:42443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:36:40.672533 359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:36:40.673686 359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:36:40.683924 366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:36:40.683984 365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:36:40.684146 359 server_base.cc:1034] running on GCE node
W20260810 22:36:40.683911 368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:36:40.685099 359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:36:40.686555 359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:36:40.687834 359 hybrid_clock.cc:648] HybridClock initialized: now 1786401400687794 us; error 72 us; skew 500 ppm
Aug 10 22:36:40 dist-test-slave-77v9 krb5kdc[342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401400, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:36:40.691411 359 init.cc:377] Logged in from keytab as kudu/127.16.233.126@KRBTEST.COM (short username kudu)
I20260810 22:36:40.692525 359 webserver.cc:460] Webserver started at http://127.16.233.126:32945/ using document root <none> and password file <none>
I20260810 22:36:40.693259 359 fs_manager.cc:359] Metadata directory not provided
I20260810 22:36:40.693339 359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:36:40.693542 359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:36:40.696024 359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/master-0/data/instance:
uuid: "84f10ccd7de840299a6c6b5a936dc470"
format_stamp: "Formatted at 2026-08-10 22:36:40 on dist-test-slave-77v9"
server_key: "b530c58bf23da5e94b75c574b67feeee"
server_key_iv: "9e811611ea6a5088cd208ce6aed8b595"
server_key_version: "encryptionkey@0"
I20260810 22:36:40.696918 359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/master-0/wal/instance:
uuid: "84f10ccd7de840299a6c6b5a936dc470"
format_stamp: "Formatted at 2026-08-10 22:36:40 on dist-test-slave-77v9"
server_key: "b530c58bf23da5e94b75c574b67feeee"
server_key_iv: "9e811611ea6a5088cd208ce6aed8b595"
server_key_version: "encryptionkey@0"
I20260810 22:36:40.701542 359 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260810 22:36:40.704423 377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:36:40.706183 359 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:36:40.706424 359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/master-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/master-0/wal
uuid: "84f10ccd7de840299a6c6b5a936dc470"
format_stamp: "Formatted at 2026-08-10 22:36:40 on dist-test-slave-77v9"
server_key: "b530c58bf23da5e94b75c574b67feeee"
server_key_iv: "9e811611ea6a5088cd208ce6aed8b595"
server_key_version: "encryptionkey@0"
I20260810 22:36:40.706620 359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:36:40.733368 359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:36:40.737519 359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:36:40.737818 359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:36:40.746618 359 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.126:42443
I20260810 22:36:40.747036 437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.126:42443 every 8 connection(s)
I20260810 22:36:40.747905 359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
I20260810 22:36:40.751119 439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:36:40.757315 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 359
I20260810 22:36:40.757476 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/master-0/wal/instance
I20260810 22:36:40.757774 17317 external_mini_cluster.cc:1428] Setting key 9f1aefa1d8178fc3615fef5e9c55c4c4
I20260810 22:36:40.757921 439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84f10ccd7de840299a6c6b5a936dc470: Bootstrap starting.
I20260810 22:36:40.760985 439 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84f10ccd7de840299a6c6b5a936dc470: Neither blocks nor log segments found. Creating new log.
I20260810 22:36:40.761997 439 log.cc:824] T 00000000000000000000000000000000 P 84f10ccd7de840299a6c6b5a936dc470: Log is configured to *not* fsync() on all Append() calls
I20260810 22:36:40.764725 439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84f10ccd7de840299a6c6b5a936dc470: No bootstrap required, opened a new log
Aug 10 22:36:40 dist-test-slave-77v9 krb5kdc[342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:36:40.769214 439 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84f10ccd7de840299a6c6b5a936dc470 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84f10ccd7de840299a6c6b5a936dc470" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 42443 } }
I20260810 22:36:40.769586 439 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84f10ccd7de840299a6c6b5a936dc470 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:36:40.769686 439 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84f10ccd7de840299a6c6b5a936dc470 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84f10ccd7de840299a6c6b5a936dc470, State: Initialized, Role: FOLLOWER
I20260810 22:36:40.770274 439 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84f10ccd7de840299a6c6b5a936dc470 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84f10ccd7de840299a6c6b5a936dc470" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 42443 } }
I20260810 22:36:40.770512 439 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84f10ccd7de840299a6c6b5a936dc470 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:36:40.770604 439 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84f10ccd7de840299a6c6b5a936dc470 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:36:40.770718 439 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84f10ccd7de840299a6c6b5a936dc470 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:36:40.772109 439 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84f10ccd7de840299a6c6b5a936dc470 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84f10ccd7de840299a6c6b5a936dc470" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 42443 } }
I20260810 22:36:40.772553 439 leader_election.cc:302] T 00000000000000000000000000000000 P 84f10ccd7de840299a6c6b5a936dc470 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84f10ccd7de840299a6c6b5a936dc470; no voters:
I20260810 22:36:40.773017 439 leader_election.cc:288] T 00000000000000000000000000000000 P 84f10ccd7de840299a6c6b5a936dc470 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:36:40.773294 444 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84f10ccd7de840299a6c6b5a936dc470 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:36:40.773578 444 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84f10ccd7de840299a6c6b5a936dc470 [term 1 LEADER]: Becoming Leader. State: Replica: 84f10ccd7de840299a6c6b5a936dc470, State: Running, Role: LEADER
I20260810 22:36:40.774010 444 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84f10ccd7de840299a6c6b5a936dc470 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84f10ccd7de840299a6c6b5a936dc470" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 42443 } }
I20260810 22:36:40.775020 439 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84f10ccd7de840299a6c6b5a936dc470 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:36:40.776873 442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:40.759425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38924 (local address 127.16.233.126:42443)
0810 22:36:40.759954 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0810 22:36:40.759990 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0810 22:36:40.760037 (+ 47us) server_negotiation.cc:407] Connection header received
0810 22:36:40.761056 (+ 1019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:36:40.761084 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:36:40.761578 (+ 494us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:36:40.761993 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:36:40.763080 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:40.764542 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:40.765600 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:40.766319 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:40.766521 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:40.769179 (+ 2658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:36:40.769187 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:36:40.769206 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:36:40.769248 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:36:40.771486 (+ 2238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:36:40.771970 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:36:40.771979 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:36:40.771988 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:36:40.772152 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:36:40.772472 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:36:40.772475 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:36:40.772477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:36:40.773110 (+ 633us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:36:40.773316 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0810 22:36:40.773700 (+ 384us) server_negotiation.cc:299] Negotiation successful
0810 22:36:40.774312 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":153,"threads_started":1}
I20260810 22:36:40.776750 445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84f10ccd7de840299a6c6b5a936dc470 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84f10ccd7de840299a6c6b5a936dc470. Latest consensus state: current_term: 1 leader_uuid: "84f10ccd7de840299a6c6b5a936dc470" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84f10ccd7de840299a6c6b5a936dc470" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 42443 } } }
I20260810 22:36:40.776966 445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84f10ccd7de840299a6c6b5a936dc470 [sys.catalog]: This master's current role is: LEADER
I20260810 22:36:40.776789 446 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84f10ccd7de840299a6c6b5a936dc470 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84f10ccd7de840299a6c6b5a936dc470" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84f10ccd7de840299a6c6b5a936dc470" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 42443 } } }
I20260810 22:36:40.777287 446 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84f10ccd7de840299a6c6b5a936dc470 [sys.catalog]: This master's current role is: LEADER
I20260810 22:36:40.778326 456 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:36:40.781656 456 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:36:40.787778 456 catalog_manager.cc:1347] Generated new cluster ID: f31b4e3c124846aeac454cc7a613b30c
I20260810 22:36:40.787889 456 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:36:40.818467 456 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:36:40.819556 456 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:36:40.824193 456 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 84f10ccd7de840299a6c6b5a936dc470: Generated new TSK 0
I20260810 22:36:40.825181 456 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.233.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab.
Entry for principal kudu/127.16.233.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab.
I20260810 22:36:40.872200 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.65:0
--local_ip_for_outbound_sockets=127.16.233.65
--webserver_interface=127.16.233.65
--webserver_port=0
--tserver_master_addrs=127.16.233.126:42443
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab
--principal=kudu/127.16.233.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:33229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:36:41.001466 467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:36:41.001791 467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:36:41.001840 467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:36:41.006291 467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:36:41.006386 467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:36:41.006520 467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.65
I20260810 22:36:41.013136 467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:33229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab
--principal=kudu/127.16.233.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.233.65
--webserver_port=0
--tserver_master_addrs=127.16.233.126:42443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.65
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:36:41.014479 467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:36:41.015726 467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 22:36:41.027105 467 server_base.cc:1034] running on GCE node
W20260810 22:36:41.027141 473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:36:41.027120 472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:36:41.027138 475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:36:41.027940 467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:36:41.028661 467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:36:41.029912 467 hybrid_clock.cc:648] HybridClock initialized: now 1786401401029882 us; error 48 us; skew 500 ppm
Aug 10 22:36:41 dist-test-slave-77v9 krb5kdc[342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401401, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:36:41.033336 467 init.cc:377] Logged in from keytab as kudu/127.16.233.65@KRBTEST.COM (short username kudu)
I20260810 22:36:41.034852 467 webserver.cc:460] Webserver started at http://127.16.233.65:37969/ using document root <none> and password file <none>
I20260810 22:36:41.035521 467 fs_manager.cc:359] Metadata directory not provided
I20260810 22:36:41.035597 467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:36:41.035796 467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:36:41.038463 467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-0/data/instance:
uuid: "7285c96084ea4dd5a35d59d5b2b47537"
format_stamp: "Formatted at 2026-08-10 22:36:41 on dist-test-slave-77v9"
server_key: "1b71f0acbd9a94be7fdc3c97ea95ca46"
server_key_iv: "c55487d61b4ce976b69716e17f40e00d"
server_key_version: "encryptionkey@0"
I20260810 22:36:41.039129 467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-0/wal/instance:
uuid: "7285c96084ea4dd5a35d59d5b2b47537"
format_stamp: "Formatted at 2026-08-10 22:36:41 on dist-test-slave-77v9"
server_key: "1b71f0acbd9a94be7fdc3c97ea95ca46"
server_key_iv: "c55487d61b4ce976b69716e17f40e00d"
server_key_version: "encryptionkey@0"
I20260810 22:36:41.043771 467 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 22:36:41.046837 485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:36:41.048666 467 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 22:36:41.049005 467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-0/wal
uuid: "7285c96084ea4dd5a35d59d5b2b47537"
format_stamp: "Formatted at 2026-08-10 22:36:41 on dist-test-slave-77v9"
server_key: "1b71f0acbd9a94be7fdc3c97ea95ca46"
server_key_iv: "c55487d61b4ce976b69716e17f40e00d"
server_key_version: "encryptionkey@0"
I20260810 22:36:41.049166 467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:36:41.063814 467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:36:41.067628 467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:36:41.067961 467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:36:41.068711 467 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:36:41.069782 467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:36:41.069854 467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:36:41.069941 467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:36:41.069978 467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:36:41.081691 467 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.65:41675
I20260810 22:36:41.081712 610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.65:41675 every 8 connection(s)
I20260810 22:36:41.082888 467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
I20260810 22:36:41.092947 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 467
I20260810 22:36:41.093114 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-0/wal/instance
I20260810 22:36:41.093469 17317 external_mini_cluster.cc:1428] Setting key 315bda8697b0be9455f616bdc0bfe06c
Aug 10 22:36:41 dist-test-slave-77v9 krb5kdc[342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401401, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.65@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:36:41.099716 442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:41.085125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:36705 (local address 127.16.233.126:42443)
0810 22:36:41.085307 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0810 22:36:41.085311 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:36:41.086437 (+ 1126us) server_negotiation.cc:407] Connection header received
0810 22:36:41.087966 (+ 1529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:36:41.087969 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:36:41.088034 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:36:41.088144 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:36:41.089517 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:41.090078 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:41.091294 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:41.091548 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:41.091667 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:41.094836 (+ 3169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:36:41.094840 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:36:41.094843 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:36:41.094873 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:36:41.096734 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:36:41.097665 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:36:41.097694 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:36:41.097699 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:36:41.097958 (+ 259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:36:41.098362 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:36:41.098367 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:36:41.098370 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:36:41.098656 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:36:41.098772 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 22:36:41.099341 (+ 569us) server_negotiation.cc:299] Negotiation successful
0810 22:36:41.099520 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260810 22:36:41.100659 613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:41.085597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:42443 (local address 127.16.233.65:36705)
0810 22:36:41.086136 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:41.086188 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 22:36:41.087720 (+ 1532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:36:41.088335 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:36:41.088342 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:36:41.088760 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:36:41.089345 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:41.089359 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:41.090225 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:41.090228 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:41.091153 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:41.091163 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:41.091736 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:41.091740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:41.091928 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:41.092609 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:36:41.092627 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:36:41.094633 (+ 2006us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:36:41.097015 (+ 2382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:36:41.097026 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:36:41.097051 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:36:41.097529 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:36:41.098078 (+ 549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:36:41.098081 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:36:41.098082 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:36:41.098256 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:36:41.098769 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:36:41.098774 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:36:41.099070 (+ 296us) client_negotiation.cc:769] Sending connection context
0810 22:36:41.099288 (+ 218us) client_negotiation.cc:240] Negotiation successful
0810 22:36:41.099686 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":138,"threads_started":1}
I20260810 22:36:41.102240 611 heartbeater.cc:347] Connected to a master server at 127.16.233.126:42443
I20260810 22:36:41.102536 611 heartbeater.cc:464] Registering TS with master...
I20260810 22:36:41.103120 611 heartbeater.cc:511] Master 127.16.233.126:42443 requested a full tablet report, sending...
I20260810 22:36:41.105016 395 ts_manager.cc:194] Registered new tserver with Master: 7285c96084ea4dd5a35d59d5b2b47537 (127.16.233.65:41675)
WARNING: no policy specified for kudu/127.16.233.66@KRBTEST.COM; defaulting to no policy
I20260810 22:36:41.106489 395 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.65@KRBTEST.COM'} at 127.16.233.65:36705
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab.
Entry for principal kudu/127.16.233.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab.
I20260810 22:36:41.128523 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.66:0
--local_ip_for_outbound_sockets=127.16.233.66
--webserver_interface=127.16.233.66
--webserver_port=0
--tserver_master_addrs=127.16.233.126:42443
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab
--principal=kudu/127.16.233.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:33229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:36:41.250850 616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:36:41.251202 616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:36:41.251250 616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:36:41.255998 616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:36:41.256086 616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:36:41.256153 616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.66
I20260810 22:36:41.262012 616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:33229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab
--principal=kudu/127.16.233.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.233.66
--webserver_port=0
--tserver_master_addrs=127.16.233.126:42443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.66
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:36:41.263340 616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:36:41.264467 616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:36:41.275233 622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:36:41.275223 623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:36:41.275300 626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:36:41.275678 616 server_base.cc:1034] running on GCE node
I20260810 22:36:41.276240 616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:36:41.277130 616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:36:41.278326 616 hybrid_clock.cc:648] HybridClock initialized: now 1786401401278285 us; error 55 us; skew 500 ppm
Aug 10 22:36:41 dist-test-slave-77v9 krb5kdc[342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401401, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:36:41.281612 616 init.cc:377] Logged in from keytab as kudu/127.16.233.66@KRBTEST.COM (short username kudu)
I20260810 22:36:41.282660 616 webserver.cc:460] Webserver started at http://127.16.233.66:34011/ using document root <none> and password file <none>
I20260810 22:36:41.283238 616 fs_manager.cc:359] Metadata directory not provided
I20260810 22:36:41.283303 616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:36:41.283488 616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:36:41.285861 616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-1/data/instance:
uuid: "10e4188e49c24a888f122a94d2349393"
format_stamp: "Formatted at 2026-08-10 22:36:41 on dist-test-slave-77v9"
server_key: "4492a97e4cb8c53a493e8b2d24fb4488"
server_key_iv: "b3b1b50c3c04d66aafa0cf8eefc69144"
server_key_version: "encryptionkey@0"
I20260810 22:36:41.286499 616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-1/wal/instance:
uuid: "10e4188e49c24a888f122a94d2349393"
format_stamp: "Formatted at 2026-08-10 22:36:41 on dist-test-slave-77v9"
server_key: "4492a97e4cb8c53a493e8b2d24fb4488"
server_key_iv: "b3b1b50c3c04d66aafa0cf8eefc69144"
server_key_version: "encryptionkey@0"
I20260810 22:36:41.290764 616 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 22:36:41.293666 633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:36:41.295076 616 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 22:36:41.295228 616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-1/wal
uuid: "10e4188e49c24a888f122a94d2349393"
format_stamp: "Formatted at 2026-08-10 22:36:41 on dist-test-slave-77v9"
server_key: "4492a97e4cb8c53a493e8b2d24fb4488"
server_key_iv: "b3b1b50c3c04d66aafa0cf8eefc69144"
server_key_version: "encryptionkey@0"
I20260810 22:36:41.295357 616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:36:41.318902 616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:36:41.322728 616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:36:41.322922 616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:36:41.323742 616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:36:41.324747 616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:36:41.324798 616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:36:41.324935 616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:36:41.324980 616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:36:41.334824 616 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.66:41115
I20260810 22:36:41.334888 762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.66:41115 every 8 connection(s)
I20260810 22:36:41.335917 616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
I20260810 22:36:41.337447 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 616
I20260810 22:36:41.337606 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-1/wal/instance
I20260810 22:36:41.337900 17317 external_mini_cluster.cc:1428] Setting key 6eb883546692ef106314a1070ed16ea2
Aug 10 22:36:41 dist-test-slave-77v9 krb5kdc[342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401401, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.66@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.233.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.67@KRBTEST.COM" created.
I20260810 22:36:41.353940 442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:41.338247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:38767 (local address 127.16.233.126:42443)
0810 22:36:41.338417 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 22:36:41.338423 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:36:41.339459 (+ 1036us) server_negotiation.cc:407] Connection header received
0810 22:36:41.341064 (+ 1605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:36:41.341068 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:36:41.341126 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:36:41.341211 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:36:41.342973 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:41.343521 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:41.345537 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:41.345820 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:41.345920 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:41.349554 (+ 3634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:36:41.349557 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:36:41.349560 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:36:41.349616 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:36:41.351255 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:36:41.351988 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:36:41.351991 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:36:41.351993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:36:41.352080 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:36:41.352538 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:36:41.352542 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:36:41.352544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:36:41.352863 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:36:41.352987 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 22:36:41.353576 (+ 589us) server_negotiation.cc:299] Negotiation successful
0810 22:36:41.353735 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 22:36:41.354985 765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:41.338774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:42443 (local address 127.16.233.66:38767)
0810 22:36:41.339300 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:41.339359 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 22:36:41.340766 (+ 1407us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:36:41.341395 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:36:41.341403 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:36:41.342016 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:36:41.342776 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:41.342799 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:41.343680 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:41.343687 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:41.345365 (+ 1678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:41.345380 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:41.346031 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:41.346039 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:41.346307 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:41.347192 (+ 885us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:36:41.347218 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:36:41.349305 (+ 2087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:36:41.351416 (+ 2111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:36:41.351424 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:36:41.351445 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:36:41.351808 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:36:41.352191 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:36:41.352194 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:36:41.352196 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:36:41.352423 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:36:41.352987 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:36:41.352996 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:36:41.353304 (+ 308us) client_negotiation.cc:769] Sending connection context
0810 22:36:41.353517 (+ 213us) client_negotiation.cc:240] Negotiation successful
0810 22:36:41.353942 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":158,"threads_started":1}
I20260810 22:36:41.356348 763 heartbeater.cc:347] Connected to a master server at 127.16.233.126:42443
I20260810 22:36:41.356637 763 heartbeater.cc:464] Registering TS with master...
I20260810 22:36:41.357339 763 heartbeater.cc:511] Master 127.16.233.126:42443 requested a full tablet report, sending...
I20260810 22:36:41.358778 395 ts_manager.cc:194] Registered new tserver with Master: 10e4188e49c24a888f122a94d2349393 (127.16.233.66:41115)
I20260810 22:36:41.359732 395 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.66@KRBTEST.COM'} at 127.16.233.66:38767
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab.
Entry for principal kudu/127.16.233.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab.
I20260810 22:36:41.373165 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.67:0
--local_ip_for_outbound_sockets=127.16.233.67
--webserver_interface=127.16.233.67
--webserver_port=0
--tserver_master_addrs=127.16.233.126:42443
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab
--principal=kudu/127.16.233.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:33229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:36:41.501799 768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:36:41.502117 768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:36:41.502164 768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:36:41.506405 768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:36:41.506506 768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:36:41.506597 768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.67
I20260810 22:36:41.512378 768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:33229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab
--principal=kudu/127.16.233.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.233.67
--webserver_port=0
--tserver_master_addrs=127.16.233.126:42443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.67
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:36:41.513691 768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:36:41.514791 768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:36:41.525756 777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:36:41.525777 775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:36:41.525776 774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:36:41.526230 768 server_base.cc:1034] running on GCE node
I20260810 22:36:41.526710 768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:36:41.527451 768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:36:41.528654 768 hybrid_clock.cc:648] HybridClock initialized: now 1786401401528620 us; error 47 us; skew 500 ppm
Aug 10 22:36:41 dist-test-slave-77v9 krb5kdc[342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401401, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:36:41.531906 768 init.cc:377] Logged in from keytab as kudu/127.16.233.67@KRBTEST.COM (short username kudu)
I20260810 22:36:41.532972 768 webserver.cc:460] Webserver started at http://127.16.233.67:42629/ using document root <none> and password file <none>
I20260810 22:36:41.533568 768 fs_manager.cc:359] Metadata directory not provided
I20260810 22:36:41.533636 768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:36:41.533834 768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:36:41.536178 768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-2/data/instance:
uuid: "89ffcbdb9f8c45c7a748bca58e869e23"
format_stamp: "Formatted at 2026-08-10 22:36:41 on dist-test-slave-77v9"
server_key: "49282af95a15dc914a459555823d5293"
server_key_iv: "0d44a7994ff90ca1fff763e838475662"
server_key_version: "encryptionkey@0"
I20260810 22:36:41.536870 768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-2/wal/instance:
uuid: "89ffcbdb9f8c45c7a748bca58e869e23"
format_stamp: "Formatted at 2026-08-10 22:36:41 on dist-test-slave-77v9"
server_key: "49282af95a15dc914a459555823d5293"
server_key_iv: "0d44a7994ff90ca1fff763e838475662"
server_key_version: "encryptionkey@0"
I20260810 22:36:41.541740 768 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260810 22:36:41.544584 784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:36:41.546203 768 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 22:36:41.546388 768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-2/wal
uuid: "89ffcbdb9f8c45c7a748bca58e869e23"
format_stamp: "Formatted at 2026-08-10 22:36:41 on dist-test-slave-77v9"
server_key: "49282af95a15dc914a459555823d5293"
server_key_iv: "0d44a7994ff90ca1fff763e838475662"
server_key_version: "encryptionkey@0"
I20260810 22:36:41.546550 768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:36:41.574150 768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:36:41.578554 768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:36:41.578809 768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:36:41.579604 768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:36:41.580564 768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:36:41.580621 768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:36:41.580667 768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:36:41.580683 768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:36:41.590687 768 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.67:46119
I20260810 22:36:41.590749 912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.67:46119 every 8 connection(s)
I20260810 22:36:41.591748 768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
I20260810 22:36:41.593070 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 768
I20260810 22:36:41.593204 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-2/wal/instance
I20260810 22:36:41.593503 17317 external_mini_cluster.cc:1428] Setting key 630200d3703ff6bb606fbf7fa81778b9
Aug 10 22:36:41 dist-test-slave-77v9 krb5kdc[342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401401, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.67@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:36:41.609606 442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:41.594317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:57291 (local address 127.16.233.126:42443)
0810 22:36:41.595878 (+ 1561us) server_negotiation.cc:206] Beginning negotiation
0810 22:36:41.595883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:36:41.595900 (+ 17us) server_negotiation.cc:407] Connection header received
0810 22:36:41.597323 (+ 1423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:36:41.597326 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:36:41.597383 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:36:41.597474 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:36:41.598910 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:41.599654 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:41.601152 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:41.601404 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:41.601516 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:41.604542 (+ 3026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:36:41.604546 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:36:41.604549 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:36:41.604581 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:36:41.606522 (+ 1941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:36:41.607344 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:36:41.607348 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:36:41.607350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:36:41.607464 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:36:41.608125 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:36:41.608129 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:36:41.608130 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:36:41.608448 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:36:41.608563 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 22:36:41.609341 (+ 778us) server_negotiation.cc:299] Negotiation successful
0810 22:36:41.609471 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1447}
I20260810 22:36:41.610494 915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:41.594782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:42443 (local address 127.16.233.67:57291)
0810 22:36:41.595445 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:41.595499 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 22:36:41.597047 (+ 1548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:36:41.597629 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:36:41.597637 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:36:41.598071 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:36:41.598660 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:41.598678 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:41.599837 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:41.599840 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:41.601012 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:41.601025 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:41.601643 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:41.601648 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:41.601893 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:41.602614 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:36:41.602633 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:36:41.604291 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:36:41.606773 (+ 2482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:36:41.606780 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:36:41.606800 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:36:41.607213 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:36:41.607592 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:36:41.607595 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:36:41.607597 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:36:41.607813 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:36:41.608572 (+ 759us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:36:41.608578 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:36:41.609073 (+ 495us) client_negotiation.cc:769] Sending connection context
0810 22:36:41.609258 (+ 185us) client_negotiation.cc:240] Negotiation successful
0810 22:36:41.609559 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":179,"threads_started":1}
I20260810 22:36:41.611743 913 heartbeater.cc:347] Connected to a master server at 127.16.233.126:42443
I20260810 22:36:41.612010 913 heartbeater.cc:464] Registering TS with master...
I20260810 22:36:41.612519 913 heartbeater.cc:511] Master 127.16.233.126:42443 requested a full tablet report, sending...
I20260810 22:36:41.613703 395 ts_manager.cc:194] Registered new tserver with Master: 89ffcbdb9f8c45c7a748bca58e869e23 (127.16.233.67:46119)
I20260810 22:36:41.614423 395 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.67@KRBTEST.COM'} at 127.16.233.67:57291
I20260810 22:36:41.621755 17317 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 22:36:41.633139 442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:41.624219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38928 (local address 127.16.233.126:42443)
0810 22:36:41.624411 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 22:36:41.624416 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:36:41.624622 (+ 206us) server_negotiation.cc:407] Connection header received
0810 22:36:41.624740 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:36:41.624742 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:36:41.624788 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:36:41.624920 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:36:41.625726 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:41.626357 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:41.627217 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:41.627474 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:41.627581 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:41.629297 (+ 1716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:36:41.629301 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:36:41.629303 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:36:41.629333 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:36:41.631031 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:36:41.631560 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:36:41.631562 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:36:41.631564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:36:41.631668 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:36:41.631972 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:36:41.631976 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:36:41.631977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:36:41.632270 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:36:41.632411 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0810 22:36:41.632690 (+ 279us) server_negotiation.cc:299] Negotiation successful
0810 22:36:41.632912 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 22:36:41.637422 395 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38928:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 22:36:41.641019 395 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:36:41.658227 928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:41.651412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.65:41675 (local address 127.0.0.1:35246)
0810 22:36:41.652692 (+ 1280us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:41.652708 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 22:36:41.652831 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:36:41.653423 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:36:41.653429 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:36:41.653452 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:36:41.653597 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:41.653608 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:41.654781 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:41.654787 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:41.656268 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:41.656278 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:41.657574 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:41.657578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:41.657694 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:41.657735 (+ 41us) client_negotiation.cc:769] Sending connection context
0810 22:36:41.657872 (+ 137us) client_negotiation.cc:240] Negotiation successful
0810 22:36:41.658018 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1219,"thread_start_us":134,"threads_started":1}
I20260810 22:36:41.658790 925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:41.651339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.67:46119 (local address 127.0.0.1:52498)
0810 22:36:41.652384 (+ 1045us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:41.652447 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0810 22:36:41.652592 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:36:41.653546 (+ 954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:36:41.653550 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:36:41.653621 (+ 71us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:36:41.653759 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:41.653767 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:41.655616 (+ 1849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:41.655623 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:41.657048 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:41.657060 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:41.658344 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:41.658348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:41.658474 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:41.658498 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 22:36:41.658570 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 22:36:41.658650 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":852,"mutex_wait_us":7,"thread_start_us":109,"threads_started":1}
I20260810 22:36:41.658806 929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:41.652229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35246 (local address 127.16.233.65:41675)
0810 22:36:41.652925 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0810 22:36:41.652933 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 22:36:41.652957 (+ 24us) server_negotiation.cc:407] Connection header received
0810 22:36:41.653041 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:36:41.653050 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:36:41.653269 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:36:41.653431 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:36:41.653734 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:41.654607 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:41.656426 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:41.657186 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:41.657311 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:41.657504 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0810 22:36:41.657943 (+ 439us) server_negotiation.cc:299] Negotiation successful
0810 22:36:41.658075 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":117,"threads_started":1}
I20260810 22:36:41.659057 930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:41.652229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33146 (local address 127.16.233.66:41115)
0810 22:36:41.652679 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0810 22:36:41.652691 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 22:36:41.653171 (+ 480us) server_negotiation.cc:407] Connection header received
0810 22:36:41.653253 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:36:41.653258 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:36:41.653435 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:36:41.653612 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:36:41.653945 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:41.655504 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:41.657267 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:41.657948 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:41.658064 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:41.658210 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 22:36:41.658766 (+ 556us) server_negotiation.cc:299] Negotiation successful
0810 22:36:41.658877 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":167,"threads_started":1}
I20260810 22:36:41.659525 926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:41.651697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.66:41115 (local address 127.0.0.1:33146)
0810 22:36:41.653012 (+ 1315us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:41.653025 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 22:36:41.653111 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:36:41.653650 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:36:41.653656 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:36:41.653686 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:36:41.653808 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:41.653815 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:41.655732 (+ 1917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:41.655736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:41.657080 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:41.657092 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:41.658282 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:41.658286 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:41.658593 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:41.658620 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 22:36:41.659256 (+ 636us) client_negotiation.cc:240] Negotiation successful
0810 22:36:41.659343 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1259,"mutex_wait_us":23,"spinlock_wait_cycles":896,"thread_start_us":76,"threads_started":1}
I20260810 22:36:41.660436 927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:41.651428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52498 (local address 127.16.233.67:46119)
0810 22:36:41.653013 (+ 1585us) server_negotiation.cc:206] Beginning negotiation
0810 22:36:41.653028 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 22:36:41.653049 (+ 21us) server_negotiation.cc:407] Connection header received
0810 22:36:41.653152 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:36:41.653158 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:36:41.653363 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:36:41.653948 (+ 585us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:36:41.654135 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:41.655430 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:41.657191 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:41.658019 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:41.658125 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:41.658268 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 22:36:41.659477 (+ 1209us) server_negotiation.cc:299] Negotiation successful
0810 22:36:41.659583 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1485,"thread_start_us":206,"threads_started":1}
I20260810 22:36:41.662078 538 tablet_service.cc:1467] Processing CreateTablet for tablet dcea0236b95d4c78abe0fe9948c2560c (DEFAULT_TABLE table=test-table [id=97494af8b662442186f321502f7df8e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:36:41.662106 686 tablet_service.cc:1467] Processing CreateTablet for tablet dcea0236b95d4c78abe0fe9948c2560c (DEFAULT_TABLE table=test-table [id=97494af8b662442186f321502f7df8e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:36:41.663305 538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcea0236b95d4c78abe0fe9948c2560c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:36:41.663311 686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcea0236b95d4c78abe0fe9948c2560c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:36:41.663228 841 tablet_service.cc:1467] Processing CreateTablet for tablet dcea0236b95d4c78abe0fe9948c2560c (DEFAULT_TABLE table=test-table [id=97494af8b662442186f321502f7df8e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:36:41.664304 841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcea0236b95d4c78abe0fe9948c2560c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:36:41.670204 932 tablet_bootstrap.cc:492] T dcea0236b95d4c78abe0fe9948c2560c P 10e4188e49c24a888f122a94d2349393: Bootstrap starting.
I20260810 22:36:41.671479 933 tablet_bootstrap.cc:492] T dcea0236b95d4c78abe0fe9948c2560c P 7285c96084ea4dd5a35d59d5b2b47537: Bootstrap starting.
I20260810 22:36:41.672390 934 tablet_bootstrap.cc:492] T dcea0236b95d4c78abe0fe9948c2560c P 89ffcbdb9f8c45c7a748bca58e869e23: Bootstrap starting.
I20260810 22:36:41.673422 932 tablet_bootstrap.cc:654] T dcea0236b95d4c78abe0fe9948c2560c P 10e4188e49c24a888f122a94d2349393: Neither blocks nor log segments found. Creating new log.
I20260810 22:36:41.673789 933 tablet_bootstrap.cc:654] T dcea0236b95d4c78abe0fe9948c2560c P 7285c96084ea4dd5a35d59d5b2b47537: Neither blocks nor log segments found. Creating new log.
I20260810 22:36:41.674475 932 log.cc:824] T dcea0236b95d4c78abe0fe9948c2560c P 10e4188e49c24a888f122a94d2349393: Log is configured to *not* fsync() on all Append() calls
I20260810 22:36:41.674698 933 log.cc:824] T dcea0236b95d4c78abe0fe9948c2560c P 7285c96084ea4dd5a35d59d5b2b47537: Log is configured to *not* fsync() on all Append() calls
I20260810 22:36:41.675210 934 tablet_bootstrap.cc:654] T dcea0236b95d4c78abe0fe9948c2560c P 89ffcbdb9f8c45c7a748bca58e869e23: Neither blocks nor log segments found. Creating new log.
I20260810 22:36:41.676122 934 log.cc:824] T dcea0236b95d4c78abe0fe9948c2560c P 89ffcbdb9f8c45c7a748bca58e869e23: Log is configured to *not* fsync() on all Append() calls
I20260810 22:36:41.677202 932 tablet_bootstrap.cc:492] T dcea0236b95d4c78abe0fe9948c2560c P 10e4188e49c24a888f122a94d2349393: No bootstrap required, opened a new log
I20260810 22:36:41.677218 933 tablet_bootstrap.cc:492] T dcea0236b95d4c78abe0fe9948c2560c P 7285c96084ea4dd5a35d59d5b2b47537: No bootstrap required, opened a new log
I20260810 22:36:41.677435 933 ts_tablet_manager.cc:1397] T dcea0236b95d4c78abe0fe9948c2560c P 7285c96084ea4dd5a35d59d5b2b47537: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 22:36:41.677435 932 ts_tablet_manager.cc:1397] T dcea0236b95d4c78abe0fe9948c2560c P 10e4188e49c24a888f122a94d2349393: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.004s
I20260810 22:36:41.678413 934 tablet_bootstrap.cc:492] T dcea0236b95d4c78abe0fe9948c2560c P 89ffcbdb9f8c45c7a748bca58e869e23: No bootstrap required, opened a new log
I20260810 22:36:41.678673 934 ts_tablet_manager.cc:1397] T dcea0236b95d4c78abe0fe9948c2560c P 89ffcbdb9f8c45c7a748bca58e869e23: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 22:36:41.682003 932 raft_consensus.cc:348] T dcea0236b95d4c78abe0fe9948c2560c P 10e4188e49c24a888f122a94d2349393 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89ffcbdb9f8c45c7a748bca58e869e23" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 46119 } } peers { permanent_uuid: "7285c96084ea4dd5a35d59d5b2b47537" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 41675 } } peers { permanent_uuid: "10e4188e49c24a888f122a94d2349393" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 41115 } }
I20260810 22:36:41.682367 932 raft_consensus.cc:374] T dcea0236b95d4c78abe0fe9948c2560c P 10e4188e49c24a888f122a94d2349393 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:36:41.682478 932 raft_consensus.cc:729] T dcea0236b95d4c78abe0fe9948c2560c P 10e4188e49c24a888f122a94d2349393 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10e4188e49c24a888f122a94d2349393, State: Initialized, Role: FOLLOWER
I20260810 22:36:41.682416 933 raft_consensus.cc:348] T dcea0236b95d4c78abe0fe9948c2560c P 7285c96084ea4dd5a35d59d5b2b47537 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89ffcbdb9f8c45c7a748bca58e869e23" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 46119 } } peers { permanent_uuid: "7285c96084ea4dd5a35d59d5b2b47537" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 41675 } } peers { permanent_uuid: "10e4188e49c24a888f122a94d2349393" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 41115 } }
I20260810 22:36:41.682679 933 raft_consensus.cc:374] T dcea0236b95d4c78abe0fe9948c2560c P 7285c96084ea4dd5a35d59d5b2b47537 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:36:41.682745 933 raft_consensus.cc:729] T dcea0236b95d4c78abe0fe9948c2560c P 7285c96084ea4dd5a35d59d5b2b47537 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7285c96084ea4dd5a35d59d5b2b47537, State: Initialized, Role: FOLLOWER
I20260810 22:36:41.683065 932 consensus_queue.cc:260] T dcea0236b95d4c78abe0fe9948c2560c P 10e4188e49c24a888f122a94d2349393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89ffcbdb9f8c45c7a748bca58e869e23" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 46119 } } peers { permanent_uuid: "7285c96084ea4dd5a35d59d5b2b47537" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 41675 } } peers { permanent_uuid: "10e4188e49c24a888f122a94d2349393" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 41115 } }
I20260810 22:36:41.683118 933 consensus_queue.cc:260] T dcea0236b95d4c78abe0fe9948c2560c P 7285c96084ea4dd5a35d59d5b2b47537 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89ffcbdb9f8c45c7a748bca58e869e23" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 46119 } } peers { permanent_uuid: "7285c96084ea4dd5a35d59d5b2b47537" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 41675 } } peers { permanent_uuid: "10e4188e49c24a888f122a94d2349393" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 41115 } }
I20260810 22:36:41.683033 934 raft_consensus.cc:348] T dcea0236b95d4c78abe0fe9948c2560c P 89ffcbdb9f8c45c7a748bca58e869e23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89ffcbdb9f8c45c7a748bca58e869e23" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 46119 } } peers { permanent_uuid: "7285c96084ea4dd5a35d59d5b2b47537" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 41675 } } peers { permanent_uuid: "10e4188e49c24a888f122a94d2349393" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 41115 } }
I20260810 22:36:41.683384 934 raft_consensus.cc:374] T dcea0236b95d4c78abe0fe9948c2560c P 89ffcbdb9f8c45c7a748bca58e869e23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:36:41.683478 934 raft_consensus.cc:729] T dcea0236b95d4c78abe0fe9948c2560c P 89ffcbdb9f8c45c7a748bca58e869e23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89ffcbdb9f8c45c7a748bca58e869e23, State: Initialized, Role: FOLLOWER
I20260810 22:36:41.683965 934 consensus_queue.cc:260] T dcea0236b95d4c78abe0fe9948c2560c P 89ffcbdb9f8c45c7a748bca58e869e23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89ffcbdb9f8c45c7a748bca58e869e23" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 46119 } } peers { permanent_uuid: "7285c96084ea4dd5a35d59d5b2b47537" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 41675 } } peers { permanent_uuid: "10e4188e49c24a888f122a94d2349393" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 41115 } }
I20260810 22:36:41.684314 933 ts_tablet_manager.cc:1428] T dcea0236b95d4c78abe0fe9948c2560c P 7285c96084ea4dd5a35d59d5b2b47537: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260810 22:36:41.684314 932 ts_tablet_manager.cc:1428] T dcea0236b95d4c78abe0fe9948c2560c P 10e4188e49c24a888f122a94d2349393: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260810 22:36:41.684717 763 heartbeater.cc:503] Master 127.16.233.126:42443 was elected leader, sending a full tablet report...
I20260810 22:36:41.684717 913 heartbeater.cc:503] Master 127.16.233.126:42443 was elected leader, sending a full tablet report...
I20260810 22:36:41.685051 934 ts_tablet_manager.cc:1428] T dcea0236b95d4c78abe0fe9948c2560c P 89ffcbdb9f8c45c7a748bca58e869e23: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260810 22:36:41.685495 611 heartbeater.cc:503] Master 127.16.233.126:42443 was elected leader, sending a full tablet report...
W20260810 22:36:41.834509 612 tablet.cc:2369] T dcea0236b95d4c78abe0fe9948c2560c P 7285c96084ea4dd5a35d59d5b2b47537: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 22:36:41.837311 764 tablet.cc:2369] T dcea0236b95d4c78abe0fe9948c2560c P 10e4188e49c24a888f122a94d2349393: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 22:36:41.843024 914 tablet.cc:2369] T dcea0236b95d4c78abe0fe9948c2560c P 89ffcbdb9f8c45c7a748bca58e869e23: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:36:41.980252 941 raft_consensus.cc:482] T dcea0236b95d4c78abe0fe9948c2560c P 89ffcbdb9f8c45c7a748bca58e869e23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:36:41.980605 941 raft_consensus.cc:504] T dcea0236b95d4c78abe0fe9948c2560c P 89ffcbdb9f8c45c7a748bca58e869e23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89ffcbdb9f8c45c7a748bca58e869e23" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 46119 } } peers { permanent_uuid: "7285c96084ea4dd5a35d59d5b2b47537" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 41675 } } peers { permanent_uuid: "10e4188e49c24a888f122a94d2349393" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 41115 } }
I20260810 22:36:41.981976 941 leader_election.cc:288] T dcea0236b95d4c78abe0fe9948c2560c P 89ffcbdb9f8c45c7a748bca58e869e23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7285c96084ea4dd5a35d59d5b2b47537 (127.16.233.65:41675), 10e4188e49c24a888f122a94d2349393 (127.16.233.66:41115)
I20260810 22:36:41.987690 930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:41.982420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:53057 (local address 127.16.233.66:41115)
0810 22:36:41.982558 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 22:36:41.982564 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:36:41.982907 (+ 343us) server_negotiation.cc:407] Connection header received
0810 22:36:41.982972 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:36:41.982975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:36:41.983030 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:36:41.983122 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:36:41.983621 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:41.984454 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:41.986293 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:41.986951 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:41.987116 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:41.987172 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 22:36:41.987470 (+ 298us) server_negotiation.cc:299] Negotiation successful
0810 22:36:41.987549 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 22:36:41.987690 929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:41.982372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:44173 (local address 127.16.233.65:41675)
0810 22:36:41.982538 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 22:36:41.982543 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:36:41.982569 (+ 26us) server_negotiation.cc:407] Connection header received
0810 22:36:41.982925 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:36:41.982929 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:36:41.982993 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:36:41.983128 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:36:41.983621 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:41.984442 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:41.986293 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:41.987013 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:41.987124 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:41.987171 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0810 22:36:41.987470 (+ 299us) server_negotiation.cc:299] Negotiation successful
0810 22:36:41.987549 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 22:36:41.987725 915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:41.982200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.65:41675 (local address 127.16.233.67:44173)
0810 22:36:41.982504 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:41.982521 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 22:36:41.982639 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:36:41.983161 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:36:41.983164 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:36:41.983206 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:36:41.983452 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:41.983461 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:41.984581 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:41.984585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:41.986117 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:41.986130 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:41.987122 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:41.987126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:41.987293 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:41.987356 (+ 63us) client_negotiation.cc:769] Sending connection context
0810 22:36:41.987445 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 22:36:41.987542 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":157,"spinlock_wait_cycles":33408}
I20260810 22:36:41.987733 942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:41.982313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.66:41115 (local address 127.16.233.67:53057)
0810 22:36:41.982783 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:41.982795 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 22:36:41.982878 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:36:41.983161 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:36:41.983164 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:36:41.983208 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:36:41.983457 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:41.983462 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:41.984581 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:41.984585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:41.986124 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:41.986130 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:41.987123 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:41.987126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:41.987299 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:41.987352 (+ 53us) client_negotiation.cc:769] Sending connection context
0810 22:36:41.987436 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 22:36:41.987542 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"mutex_wait_us":340,"spinlock_wait_cycles":7424,"thread_start_us":138,"threads_started":1}
I20260810 22:36:41.988571 559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea0236b95d4c78abe0fe9948c2560c" candidate_uuid: "89ffcbdb9f8c45c7a748bca58e869e23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7285c96084ea4dd5a35d59d5b2b47537" is_pre_election: true
I20260810 22:36:41.988726 709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea0236b95d4c78abe0fe9948c2560c" candidate_uuid: "89ffcbdb9f8c45c7a748bca58e869e23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10e4188e49c24a888f122a94d2349393" is_pre_election: true
I20260810 22:36:41.988976 559 raft_consensus.cc:2454] T dcea0236b95d4c78abe0fe9948c2560c P 7285c96084ea4dd5a35d59d5b2b47537 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89ffcbdb9f8c45c7a748bca58e869e23 in term 0.
I20260810 22:36:41.989102 709 raft_consensus.cc:2454] T dcea0236b95d4c78abe0fe9948c2560c P 10e4188e49c24a888f122a94d2349393 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89ffcbdb9f8c45c7a748bca58e869e23 in term 0.
I20260810 22:36:41.989604 786 leader_election.cc:302] T dcea0236b95d4c78abe0fe9948c2560c P 89ffcbdb9f8c45c7a748bca58e869e23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 10e4188e49c24a888f122a94d2349393, 7285c96084ea4dd5a35d59d5b2b47537, 89ffcbdb9f8c45c7a748bca58e869e23; no voters:
I20260810 22:36:41.989920 941 raft_consensus.cc:2781] T dcea0236b95d4c78abe0fe9948c2560c P 89ffcbdb9f8c45c7a748bca58e869e23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:36:41.990033 941 raft_consensus.cc:482] T dcea0236b95d4c78abe0fe9948c2560c P 89ffcbdb9f8c45c7a748bca58e869e23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:36:41.990099 941 raft_consensus.cc:3037] T dcea0236b95d4c78abe0fe9948c2560c P 89ffcbdb9f8c45c7a748bca58e869e23 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:36:41.991232 941 raft_consensus.cc:504] T dcea0236b95d4c78abe0fe9948c2560c P 89ffcbdb9f8c45c7a748bca58e869e23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89ffcbdb9f8c45c7a748bca58e869e23" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 46119 } } peers { permanent_uuid: "7285c96084ea4dd5a35d59d5b2b47537" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 41675 } } peers { permanent_uuid: "10e4188e49c24a888f122a94d2349393" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 41115 } }
I20260810 22:36:41.991678 941 leader_election.cc:288] T dcea0236b95d4c78abe0fe9948c2560c P 89ffcbdb9f8c45c7a748bca58e869e23 [CANDIDATE]: Term 1 election: Requested vote from peers 7285c96084ea4dd5a35d59d5b2b47537 (127.16.233.65:41675), 10e4188e49c24a888f122a94d2349393 (127.16.233.66:41115)
I20260810 22:36:41.992223 709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea0236b95d4c78abe0fe9948c2560c" candidate_uuid: "89ffcbdb9f8c45c7a748bca58e869e23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10e4188e49c24a888f122a94d2349393"
I20260810 22:36:41.992223 559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea0236b95d4c78abe0fe9948c2560c" candidate_uuid: "89ffcbdb9f8c45c7a748bca58e869e23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7285c96084ea4dd5a35d59d5b2b47537"
I20260810 22:36:41.992374 559 raft_consensus.cc:3037] T dcea0236b95d4c78abe0fe9948c2560c P 7285c96084ea4dd5a35d59d5b2b47537 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:36:41.992374 709 raft_consensus.cc:3037] T dcea0236b95d4c78abe0fe9948c2560c P 10e4188e49c24a888f122a94d2349393 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:36:41.993898 709 raft_consensus.cc:2454] T dcea0236b95d4c78abe0fe9948c2560c P 10e4188e49c24a888f122a94d2349393 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89ffcbdb9f8c45c7a748bca58e869e23 in term 1.
I20260810 22:36:41.993898 559 raft_consensus.cc:2454] T dcea0236b95d4c78abe0fe9948c2560c P 7285c96084ea4dd5a35d59d5b2b47537 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89ffcbdb9f8c45c7a748bca58e869e23 in term 1.
I20260810 22:36:41.994287 785 leader_election.cc:302] T dcea0236b95d4c78abe0fe9948c2560c P 89ffcbdb9f8c45c7a748bca58e869e23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10e4188e49c24a888f122a94d2349393, 89ffcbdb9f8c45c7a748bca58e869e23; no voters:
I20260810 22:36:41.994496 941 raft_consensus.cc:2781] T dcea0236b95d4c78abe0fe9948c2560c P 89ffcbdb9f8c45c7a748bca58e869e23 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:36:41.994791 941 raft_consensus.cc:686] T dcea0236b95d4c78abe0fe9948c2560c P 89ffcbdb9f8c45c7a748bca58e869e23 [term 1 LEADER]: Becoming Leader. State: Replica: 89ffcbdb9f8c45c7a748bca58e869e23, State: Running, Role: LEADER
I20260810 22:36:41.995249 941 consensus_queue.cc:237] T dcea0236b95d4c78abe0fe9948c2560c P 89ffcbdb9f8c45c7a748bca58e869e23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89ffcbdb9f8c45c7a748bca58e869e23" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 46119 } } peers { permanent_uuid: "7285c96084ea4dd5a35d59d5b2b47537" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 41675 } } peers { permanent_uuid: "10e4188e49c24a888f122a94d2349393" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 41115 } }
I20260810 22:36:41.999018 395 catalog_manager.cc:5495] T dcea0236b95d4c78abe0fe9948c2560c P 89ffcbdb9f8c45c7a748bca58e869e23 reported cstate change: term changed from 0 to 1, leader changed from <none> to 89ffcbdb9f8c45c7a748bca58e869e23 (127.16.233.67). New cstate: current_term: 1 leader_uuid: "89ffcbdb9f8c45c7a748bca58e869e23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89ffcbdb9f8c45c7a748bca58e869e23" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 46119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7285c96084ea4dd5a35d59d5b2b47537" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 41675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10e4188e49c24a888f122a94d2349393" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 41115 } health_report { overall_health: UNKNOWN } } }
I20260810 22:36:42.027344 927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:42.023368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52500 (local address 127.16.233.67:46119)
0810 22:36:42.023544 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0810 22:36:42.023549 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:36:42.023567 (+ 18us) server_negotiation.cc:407] Connection header received
0810 22:36:42.023660 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:36:42.023662 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:36:42.023721 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:36:42.023845 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:36:42.024128 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:42.024647 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:42.025677 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:42.025905 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:42.026039 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:42.026402 (+ 363us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:36:42.026889 (+ 487us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:36:42.027039 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0810 22:36:42.027108 (+ 69us) server_negotiation.cc:299] Negotiation successful
0810 22:36:42.027177 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 22:36:42.034274 559 raft_consensus.cc:1261] T dcea0236b95d4c78abe0fe9948c2560c P 7285c96084ea4dd5a35d59d5b2b47537 [term 1 FOLLOWER]: Refusing update from remote peer 89ffcbdb9f8c45c7a748bca58e869e23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:36:42.034274 709 raft_consensus.cc:1261] T dcea0236b95d4c78abe0fe9948c2560c P 10e4188e49c24a888f122a94d2349393 [term 1 FOLLOWER]: Refusing update from remote peer 89ffcbdb9f8c45c7a748bca58e869e23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:36:42.035136 941 consensus_queue.cc:1035] T dcea0236b95d4c78abe0fe9948c2560c P 89ffcbdb9f8c45c7a748bca58e869e23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7285c96084ea4dd5a35d59d5b2b47537" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 41675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:36:42.035362 943 consensus_queue.cc:1035] T dcea0236b95d4c78abe0fe9948c2560c P 89ffcbdb9f8c45c7a748bca58e869e23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10e4188e49c24a888f122a94d2349393" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 41115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:36:42.053932 395 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:36:42.054196 395 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:36:42.057256 395 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 84f10ccd7de840299a6c6b5a936dc470: Sending DeleteTablet for 3 replicas of tablet dcea0236b95d4c78abe0fe9948c2560c
I20260810 22:36:42.058323 538 tablet_service.cc:1514] Processing DeleteTablet for tablet dcea0236b95d4c78abe0fe9948c2560c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:36:42 UTC) from {username='kudu', principal='kudu/127.16.233.126@KRBTEST.COM'} at 127.0.0.1:35246
I20260810 22:36:42.058388 686 tablet_service.cc:1514] Processing DeleteTablet for tablet dcea0236b95d4c78abe0fe9948c2560c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:36:42 UTC) from {username='kudu', principal='kudu/127.16.233.126@KRBTEST.COM'} at 127.0.0.1:33146
I20260810 22:36:42.058952 956 tablet_replica.cc:330] T dcea0236b95d4c78abe0fe9948c2560c P 7285c96084ea4dd5a35d59d5b2b47537: stopping tablet replica
I20260810 22:36:42.059267 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 467
I20260810 22:36:42.059234 841 tablet_service.cc:1514] Processing DeleteTablet for tablet dcea0236b95d4c78abe0fe9948c2560c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:36:42 UTC) from {username='kudu', principal='kudu/127.16.233.126@KRBTEST.COM'} at 127.0.0.1:52498
I20260810 22:36:42.064756 957 tablet_replica.cc:330] T dcea0236b95d4c78abe0fe9948c2560c P 10e4188e49c24a888f122a94d2349393: stopping tablet replica
I20260810 22:36:42.065302 957 raft_consensus.cc:2229] T dcea0236b95d4c78abe0fe9948c2560c P 10e4188e49c24a888f122a94d2349393 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:36:42.065773 957 raft_consensus.cc:2258] T dcea0236b95d4c78abe0fe9948c2560c P 10e4188e49c24a888f122a94d2349393 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:36:42.066860 958 tablet_replica.cc:330] T dcea0236b95d4c78abe0fe9948c2560c P 89ffcbdb9f8c45c7a748bca58e869e23: stopping tablet replica
I20260810 22:36:42.067256 958 raft_consensus.cc:2229] T dcea0236b95d4c78abe0fe9948c2560c P 89ffcbdb9f8c45c7a748bca58e869e23 [term 1 LEADER]: Raft consensus shutting down.
I20260810 22:36:42.067729 958 raft_consensus.cc:2258] T dcea0236b95d4c78abe0fe9948c2560c P 89ffcbdb9f8c45c7a748bca58e869e23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:36:42.067732 957 ts_tablet_manager.cc:1905] T dcea0236b95d4c78abe0fe9948c2560c P 10e4188e49c24a888f122a94d2349393: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:36:42.069523 958 ts_tablet_manager.cc:1905] T dcea0236b95d4c78abe0fe9948c2560c P 89ffcbdb9f8c45c7a748bca58e869e23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:36:42.071045 957 ts_tablet_manager.cc:1918] T dcea0236b95d4c78abe0fe9948c2560c P 10e4188e49c24a888f122a94d2349393: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:36:42.071288 957 log.cc:1196] T dcea0236b95d4c78abe0fe9948c2560c P 10e4188e49c24a888f122a94d2349393: Deleting WAL directory at /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-1/wal/wals/dcea0236b95d4c78abe0fe9948c2560c
I20260810 22:36:42.071746 957 ts_tablet_manager.cc:1939] T dcea0236b95d4c78abe0fe9948c2560c P 10e4188e49c24a888f122a94d2349393: Deleting consensus metadata
W20260810 22:36:42.072427 379 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.233.65:41675 (error 108)
I20260810 22:36:42.072798 958 ts_tablet_manager.cc:1918] T dcea0236b95d4c78abe0fe9948c2560c P 89ffcbdb9f8c45c7a748bca58e869e23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:36:42.072925 958 log.cc:1196] T dcea0236b95d4c78abe0fe9948c2560c P 89ffcbdb9f8c45c7a748bca58e869e23: Deleting WAL directory at /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-2/wal/wals/dcea0236b95d4c78abe0fe9948c2560c
I20260810 22:36:42.073253 958 ts_tablet_manager.cc:1939] T dcea0236b95d4c78abe0fe9948c2560c P 89ffcbdb9f8c45c7a748bca58e869e23: Deleting consensus metadata
I20260810 22:36:42.073405 378 catalog_manager.cc:4841] TS 10e4188e49c24a888f122a94d2349393 (127.16.233.66:41115): tablet dcea0236b95d4c78abe0fe9948c2560c (table test-table [id=97494af8b662442186f321502f7df8e3]) successfully deleted
I20260810 22:36:42.074259 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 616
I20260810 22:36:42.074285 926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:42.073959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.65:41675 (local address 127.0.0.1:35258)
0810 22:36:42.074099 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:42.074183 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.65:41675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 22:36:42.076779 379 catalog_manager.cc:4584] TS 7285c96084ea4dd5a35d59d5b2b47537 (127.16.233.65:41675): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dcea0236b95d4c78abe0fe9948c2560c: Network error: Client connection negotiation failed: client connection to 127.16.233.65:41675: connect: Connection refused (error 111)
I20260810 22:36:42.085350 381 catalog_manager.cc:4841] TS 89ffcbdb9f8c45c7a748bca58e869e23 (127.16.233.67:46119): tablet dcea0236b95d4c78abe0fe9948c2560c (table test-table [id=97494af8b662442186f321502f7df8e3]) successfully deleted
I20260810 22:36:42.090523 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 768
I20260810 22:36:42.098845 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 359
I20260810 22:36:42.110131 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:42443
--webserver_interface=127.16.233.126
--webserver_port=32945
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab
--principal=kudu/127.16.233.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:33229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.126:42443
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:36:42.233639 960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:36:42.233995 960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:36:42.234083 960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:36:42.238444 960 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:36:42.238562 960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:36:42.238600 960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:36:42.238646 960 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:36:42.238680 960 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:36:42.244508 960 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:33229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.16.233.126:42443
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab
--principal=kudu/127.16.233.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:42443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.126
--webserver_port=32945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:36:42.245882 960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:36:42.247056 960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:36:42.263053 968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:36:42.263053 967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:36:42.263053 971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:36:42.263383 960 server_base.cc:1034] running on GCE node
I20260810 22:36:42.264230 960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:36:42.265658 960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:36:42.266873 960 hybrid_clock.cc:648] HybridClock initialized: now 1786401402266859 us; error 36 us; skew 500 ppm
Aug 10 22:36:42 dist-test-slave-77v9 krb5kdc[342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401402, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:36:42.270427 960 init.cc:377] Logged in from keytab as kudu/127.16.233.126@KRBTEST.COM (short username kudu)
I20260810 22:36:42.271605 960 webserver.cc:460] Webserver started at http://127.16.233.126:32945/ using document root <none> and password file <none>
I20260810 22:36:42.272300 960 fs_manager.cc:359] Metadata directory not provided
I20260810 22:36:42.272449 960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:36:42.277460 960 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 22:36:42.280260 979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:36:42.282018 960 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:36:42.282228 960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/master-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/master-0/wal
uuid: "84f10ccd7de840299a6c6b5a936dc470"
format_stamp: "Formatted at 2026-08-10 22:36:40 on dist-test-slave-77v9"
server_key: "b530c58bf23da5e94b75c574b67feeee"
server_key_iv: "9e811611ea6a5088cd208ce6aed8b595"
server_key_version: "encryptionkey@0"
I20260810 22:36:42.283018 960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:36:42.297441 960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:36:42.301445 960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:36:42.301702 960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:36:42.311221 960 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.126:42443
I20260810 22:36:42.311604 1037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.126:42443 every 8 connection(s)
I20260810 22:36:42.312629 960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
I20260810 22:36:42.317667 1038 sys_catalog.cc:263] Verifying existing consensus state
I20260810 22:36:42.319386 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 960
I20260810 22:36:42.319700 1038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84f10ccd7de840299a6c6b5a936dc470: Bootstrap starting.
I20260810 22:36:42.320014 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.65:41675
--local_ip_for_outbound_sockets=127.16.233.65
--tserver_master_addrs=127.16.233.126:42443
--webserver_port=37969
--webserver_interface=127.16.233.65
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab
--principal=kudu/127.16.233.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:33229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 22:36:42.343848 1038 log.cc:824] T 00000000000000000000000000000000 P 84f10ccd7de840299a6c6b5a936dc470: Log is configured to *not* fsync() on all Append() calls
I20260810 22:36:42.353889 1038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84f10ccd7de840299a6c6b5a936dc470: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 22:36:42.354406 1038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84f10ccd7de840299a6c6b5a936dc470: Bootstrap complete.
I20260810 22:36:42.359071 1038 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84f10ccd7de840299a6c6b5a936dc470 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84f10ccd7de840299a6c6b5a936dc470" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 42443 } }
I20260810 22:36:42.359525 1038 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84f10ccd7de840299a6c6b5a936dc470 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84f10ccd7de840299a6c6b5a936dc470, State: Initialized, Role: FOLLOWER
I20260810 22:36:42.360195 1038 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84f10ccd7de840299a6c6b5a936dc470 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84f10ccd7de840299a6c6b5a936dc470" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 42443 } }
I20260810 22:36:42.360340 1038 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84f10ccd7de840299a6c6b5a936dc470 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:36:42.360447 1038 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84f10ccd7de840299a6c6b5a936dc470 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:36:42.360584 1038 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84f10ccd7de840299a6c6b5a936dc470 [term 1 FOLLOWER]: Advancing to term 2
I20260810 22:36:42.363003 1038 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84f10ccd7de840299a6c6b5a936dc470 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84f10ccd7de840299a6c6b5a936dc470" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 42443 } }
I20260810 22:36:42.363423 1038 leader_election.cc:302] T 00000000000000000000000000000000 P 84f10ccd7de840299a6c6b5a936dc470 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84f10ccd7de840299a6c6b5a936dc470; no voters:
I20260810 22:36:42.363842 1038 leader_election.cc:288] T 00000000000000000000000000000000 P 84f10ccd7de840299a6c6b5a936dc470 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 22:36:42.363945 1043 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84f10ccd7de840299a6c6b5a936dc470 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 22:36:42.364168 1043 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84f10ccd7de840299a6c6b5a936dc470 [term 2 LEADER]: Becoming Leader. State: Replica: 84f10ccd7de840299a6c6b5a936dc470, State: Running, Role: LEADER
I20260810 22:36:42.364495 1043 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84f10ccd7de840299a6c6b5a936dc470 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84f10ccd7de840299a6c6b5a936dc470" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 42443 } }
I20260810 22:36:42.365233 1038 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84f10ccd7de840299a6c6b5a936dc470 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:36:42.366050 1044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84f10ccd7de840299a6c6b5a936dc470 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "84f10ccd7de840299a6c6b5a936dc470" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84f10ccd7de840299a6c6b5a936dc470" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 42443 } } }
I20260810 22:36:42.366189 1044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84f10ccd7de840299a6c6b5a936dc470 [sys.catalog]: This master's current role is: LEADER
I20260810 22:36:42.366549 1045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84f10ccd7de840299a6c6b5a936dc470 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84f10ccd7de840299a6c6b5a936dc470. Latest consensus state: current_term: 2 leader_uuid: "84f10ccd7de840299a6c6b5a936dc470" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84f10ccd7de840299a6c6b5a936dc470" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 42443 } } }
I20260810 22:36:42.366705 1045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84f10ccd7de840299a6c6b5a936dc470 [sys.catalog]: This master's current role is: LEADER
I20260810 22:36:42.370733 1062 catalog_manager.cc:1259] Loaded cluster ID: f31b4e3c124846aeac454cc7a613b30c
I20260810 22:36:42.370813 1062 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 84f10ccd7de840299a6c6b5a936dc470: loading cluster ID for follower catalog manager: success
W20260810 22:36:42.371816 1062 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 84f10ccd7de840299a6c6b5a936dc470: acquiring CA information for follower catalog manager: Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG
W20260810 22:36:42.371913 1062 catalog_manager.cc:873] Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG: failed to prepare follower catalog manager, will retry
I20260810 22:36:42.371984 1053 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:36:42.374837 1053 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:36:42.375540 1053 catalog_manager.cc:1259] Loaded cluster ID: f31b4e3c124846aeac454cc7a613b30c
I20260810 22:36:42.375612 1053 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:36:42.377070 1053 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260810 22:36:42.382067 1053 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260810 22:36:42.388743 1053 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:36:42.389801 1053 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 84f10ccd7de840299a6c6b5a936dc470: Loaded TSK: 0
I20260810 22:36:42.391224 1053 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 22:36:42.461206 1040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:36:42.461508 1040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:36:42.461560 1040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:36:42.466253 1040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:36:42.466357 1040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:36:42.466482 1040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.65
I20260810 22:36:42.472970 1040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:33229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab
--principal=kudu/127.16.233.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.65:41675
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.233.65
--webserver_port=37969
--tserver_master_addrs=127.16.233.126:42443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.65
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:36:42.474267 1040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:36:42.475517 1040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:36:42.490248 1072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:36:42.490272 1074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:36:42.490296 1071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:36:42.490855 1040 server_base.cc:1034] running on GCE node
I20260810 22:36:42.491312 1040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:36:42.492108 1040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:36:42.493287 1040 hybrid_clock.cc:648] HybridClock initialized: now 1786401402493273 us; error 33 us; skew 500 ppm
Aug 10 22:36:42 dist-test-slave-77v9 krb5kdc[342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401402, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:36:42.496438 1040 init.cc:377] Logged in from keytab as kudu/127.16.233.65@KRBTEST.COM (short username kudu)
I20260810 22:36:42.497627 1040 webserver.cc:460] Webserver started at http://127.16.233.65:37969/ using document root <none> and password file <none>
I20260810 22:36:42.498356 1040 fs_manager.cc:359] Metadata directory not provided
I20260810 22:36:42.498454 1040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:36:42.503041 1040 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 22:36:42.505813 1084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:36:42.507405 1040 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:36:42.507587 1040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-0/wal
uuid: "7285c96084ea4dd5a35d59d5b2b47537"
format_stamp: "Formatted at 2026-08-10 22:36:41 on dist-test-slave-77v9"
server_key: "1b71f0acbd9a94be7fdc3c97ea95ca46"
server_key_iv: "c55487d61b4ce976b69716e17f40e00d"
server_key_version: "encryptionkey@0"
I20260810 22:36:42.508227 1040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:36:42.534644 1040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:36:42.538503 1040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:36:42.538754 1040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:36:42.539484 1040 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:36:42.540830 1092 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 22:36:42.545681 1040 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 22:36:42.545794 1040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260810 22:36:42.545914 1040 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 22:36:42.548043 1040 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 22:36:42.548110 1040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260810 22:36:42.548605 1092 tablet_bootstrap.cc:492] T dcea0236b95d4c78abe0fe9948c2560c P 7285c96084ea4dd5a35d59d5b2b47537: Bootstrap starting.
I20260810 22:36:42.560496 1040 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.65:41675
I20260810 22:36:42.560549 1221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.65:41675 every 8 connection(s)
I20260810 22:36:42.561633 1040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
I20260810 22:36:42.570850 1092 log.cc:824] T dcea0236b95d4c78abe0fe9948c2560c P 7285c96084ea4dd5a35d59d5b2b47537: Log is configured to *not* fsync() on all Append() calls
I20260810 22:36:42.571151 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 1040
I20260810 22:36:42.571656 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.66:41115
--local_ip_for_outbound_sockets=127.16.233.66
--tserver_master_addrs=127.16.233.126:42443
--webserver_port=34011
--webserver_interface=127.16.233.66
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab
--principal=kudu/127.16.233.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:33229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 22:36:42 dist-test-slave-77v9 krb5kdc[342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401402, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.65@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:36:42.580664 1092 tablet_bootstrap.cc:492] T dcea0236b95d4c78abe0fe9948c2560c P 7285c96084ea4dd5a35d59d5b2b47537: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 22:36:42.581178 1092 tablet_bootstrap.cc:492] T dcea0236b95d4c78abe0fe9948c2560c P 7285c96084ea4dd5a35d59d5b2b47537: Bootstrap complete.
I20260810 22:36:42.582747 1092 ts_tablet_manager.cc:1397] T dcea0236b95d4c78abe0fe9948c2560c P 7285c96084ea4dd5a35d59d5b2b47537: Time spent bootstrapping tablet: real 0.034s user 0.024s sys 0.008s
I20260810 22:36:42.584262 1224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:42.564181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:46877 (local address 127.16.233.126:42443)
0810 22:36:42.564619 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0810 22:36:42.564648 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0810 22:36:42.565139 (+ 491us) server_negotiation.cc:407] Connection header received
0810 22:36:42.566894 (+ 1755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:36:42.566915 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:36:42.567407 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:36:42.567808 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:36:42.569078 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:42.570528 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:42.572578 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:42.573309 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:42.573506 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:42.577116 (+ 3610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:36:42.577124 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:36:42.577141 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:36:42.577185 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:36:42.579670 (+ 2485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:36:42.580558 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:36:42.580565 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:36:42.580570 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:36:42.580801 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:36:42.581394 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:36:42.581398 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:36:42.581399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:36:42.581939 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:36:42.582087 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 22:36:42.582999 (+ 912us) server_negotiation.cc:299] Negotiation successful
0810 22:36:42.583346 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":155,"threads_started":1}
I20260810 22:36:42.584329 1225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:42.564134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:42443 (local address 127.16.233.65:46877)
0810 22:36:42.564909 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:42.564977 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0810 22:36:42.566161 (+ 1184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:36:42.567918 (+ 1757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:36:42.567926 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:36:42.568319 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:36:42.568800 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:42.568888 (+ 88us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:42.570840 (+ 1952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:42.570844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:42.572399 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:42.572418 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:42.573510 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:42.573516 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:42.573770 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:42.574585 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:36:42.574609 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:36:42.576762 (+ 2153us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:36:42.579856 (+ 3094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:36:42.579866 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:36:42.579881 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:36:42.580356 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:36:42.580965 (+ 609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:36:42.580972 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:36:42.580977 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:36:42.581196 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:36:42.582104 (+ 908us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:36:42.582114 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:36:42.582496 (+ 382us) client_negotiation.cc:769] Sending connection context
0810 22:36:42.582737 (+ 241us) client_negotiation.cc:240] Negotiation successful
0810 22:36:42.583176 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":165,"threads_started":1}
I20260810 22:36:42.587113 1222 heartbeater.cc:347] Connected to a master server at 127.16.233.126:42443
I20260810 22:36:42.587371 1222 heartbeater.cc:464] Registering TS with master...
I20260810 22:36:42.586915 1092 raft_consensus.cc:348] T dcea0236b95d4c78abe0fe9948c2560c P 7285c96084ea4dd5a35d59d5b2b47537 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89ffcbdb9f8c45c7a748bca58e869e23" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 46119 } } peers { permanent_uuid: "7285c96084ea4dd5a35d59d5b2b47537" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 41675 } } peers { permanent_uuid: "10e4188e49c24a888f122a94d2349393" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 41115 } }
I20260810 22:36:42.587548 1092 raft_consensus.cc:729] T dcea0236b95d4c78abe0fe9948c2560c P 7285c96084ea4dd5a35d59d5b2b47537 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7285c96084ea4dd5a35d59d5b2b47537, State: Initialized, Role: FOLLOWER
I20260810 22:36:42.588001 1222 heartbeater.cc:511] Master 127.16.233.126:42443 requested a full tablet report, sending...
I20260810 22:36:42.588063 1092 consensus_queue.cc:260] T dcea0236b95d4c78abe0fe9948c2560c P 7285c96084ea4dd5a35d59d5b2b47537 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89ffcbdb9f8c45c7a748bca58e869e23" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 46119 } } peers { permanent_uuid: "7285c96084ea4dd5a35d59d5b2b47537" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 41675 } } peers { permanent_uuid: "10e4188e49c24a888f122a94d2349393" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 41115 } }
I20260810 22:36:42.589268 1092 ts_tablet_manager.cc:1428] T dcea0236b95d4c78abe0fe9948c2560c P 7285c96084ea4dd5a35d59d5b2b47537: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 22:36:42.591022 997 ts_manager.cc:194] Registered new tserver with Master: 7285c96084ea4dd5a35d59d5b2b47537 (127.16.233.65:41675)
I20260810 22:36:42.594857 997 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.65@KRBTEST.COM'} at 127.16.233.65:46877
I20260810 22:36:42.598120 1222 heartbeater.cc:503] Master 127.16.233.126:42443 was elected leader, sending a full tablet report...
I20260810 22:36:42.602085 1229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:42.594356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.65:41675 (local address 127.0.0.1:35272)
0810 22:36:42.595891 (+ 1535us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:42.595947 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 22:36:42.596099 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:36:42.597414 (+ 1315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:36:42.597420 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:36:42.597449 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:36:42.597609 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:42.597617 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:42.598878 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:42.598885 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:42.600377 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:42.600389 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:42.601397 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:42.601401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:42.601543 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:42.601593 (+ 50us) client_negotiation.cc:769] Sending connection context
0810 22:36:42.601765 (+ 172us) client_negotiation.cc:240] Negotiation successful
0810 22:36:42.601927 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1312,"thread_start_us":108,"threads_started":1}
I20260810 22:36:42.602085 1230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:42.594480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35272 (local address 127.16.233.65:41675)
0810 22:36:42.596365 (+ 1885us) server_negotiation.cc:206] Beginning negotiation
0810 22:36:42.596374 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 22:36:42.596394 (+ 20us) server_negotiation.cc:407] Connection header received
0810 22:36:42.596455 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:36:42.596459 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:36:42.597164 (+ 705us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:36:42.597414 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:36:42.597752 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:42.598744 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:42.600525 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:42.601226 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:42.601383 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:42.601551 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0810 22:36:42.601791 (+ 240us) server_negotiation.cc:299] Negotiation successful
0810 22:36:42.601905 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1790,"thread_start_us":123,"threads_started":1}
I20260810 22:36:42.603276 1142 tablet_service.cc:1514] Processing DeleteTablet for tablet dcea0236b95d4c78abe0fe9948c2560c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:36:42 UTC) from {username='kudu', principal='kudu/127.16.233.126@KRBTEST.COM'} at 127.0.0.1:35272
I20260810 22:36:42.604411 1231 tablet_replica.cc:330] stopping tablet replica
I20260810 22:36:42.604779 1231 raft_consensus.cc:2229] T dcea0236b95d4c78abe0fe9948c2560c P 7285c96084ea4dd5a35d59d5b2b47537 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:36:42.605122 1231 raft_consensus.cc:2258] T dcea0236b95d4c78abe0fe9948c2560c P 7285c96084ea4dd5a35d59d5b2b47537 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:36:42.606302 1231 ts_tablet_manager.cc:1905] T dcea0236b95d4c78abe0fe9948c2560c P 7285c96084ea4dd5a35d59d5b2b47537: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:36:42.610697 1231 ts_tablet_manager.cc:1918] T dcea0236b95d4c78abe0fe9948c2560c P 7285c96084ea4dd5a35d59d5b2b47537: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:36:42.610898 1231 log.cc:1196] T dcea0236b95d4c78abe0fe9948c2560c P 7285c96084ea4dd5a35d59d5b2b47537: Deleting WAL directory at /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-0/wal/wals/dcea0236b95d4c78abe0fe9948c2560c
I20260810 22:36:42.611241 1231 ts_tablet_manager.cc:1939] T dcea0236b95d4c78abe0fe9948c2560c P 7285c96084ea4dd5a35d59d5b2b47537: Deleting consensus metadata
I20260810 22:36:42.613493 981 catalog_manager.cc:4841] TS 7285c96084ea4dd5a35d59d5b2b47537 (127.16.233.65:41675): tablet dcea0236b95d4c78abe0fe9948c2560c (table test-table [id=97494af8b662442186f321502f7df8e3]) successfully deleted
W20260810 22:36:42.708530 1227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:36:42.708926 1227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:36:42.708978 1227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:36:42.713668 1227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:36:42.713770 1227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:36:42.713840 1227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.66
I20260810 22:36:42.719782 1227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:33229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab
--principal=kudu/127.16.233.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.66:41115
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.233.66
--webserver_port=34011
--tserver_master_addrs=127.16.233.126:42443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.66
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:36:42.721146 1227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:36:42.722589 1227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:36:42.733680 1236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:36:42.733845 1227 server_base.cc:1034] running on GCE node
W20260810 22:36:42.733670 1240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:36:42.733697 1237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:36:42.734575 1227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:36:42.735404 1227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:36:42.736649 1227 hybrid_clock.cc:648] HybridClock initialized: now 1786401402736622 us; error 39 us; skew 500 ppm
Aug 10 22:36:42 dist-test-slave-77v9 krb5kdc[342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401402, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:36:42.740067 1227 init.cc:377] Logged in from keytab as kudu/127.16.233.66@KRBTEST.COM (short username kudu)
I20260810 22:36:42.741247 1227 webserver.cc:460] Webserver started at http://127.16.233.66:34011/ using document root <none> and password file <none>
I20260810 22:36:42.741873 1227 fs_manager.cc:359] Metadata directory not provided
I20260810 22:36:42.742002 1227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:36:42.746742 1227 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 22:36:42.749380 1247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:36:42.750926 1227 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:36:42.751111 1227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-1/wal
uuid: "10e4188e49c24a888f122a94d2349393"
format_stamp: "Formatted at 2026-08-10 22:36:41 on dist-test-slave-77v9"
server_key: "4492a97e4cb8c53a493e8b2d24fb4488"
server_key_iv: "b3b1b50c3c04d66aafa0cf8eefc69144"
server_key_version: "encryptionkey@0"
I20260810 22:36:42.751796 1227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:36:42.770450 1227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:36:42.774572 1227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:36:42.774837 1227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:36:42.775533 1227 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:36:42.776490 1227 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:36:42.776569 1227 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:36:42.776645 1227 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:36:42.776690 1227 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:36:42.787844 1227 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.66:41115
I20260810 22:36:42.787876 1381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.66:41115 every 8 connection(s)
I20260810 22:36:42.789108 1227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
I20260810 22:36:42.793581 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 1227
I20260810 22:36:42.794075 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.67:46119
--local_ip_for_outbound_sockets=127.16.233.67
--tserver_master_addrs=127.16.233.126:42443
--webserver_port=42629
--webserver_interface=127.16.233.67
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab
--principal=kudu/127.16.233.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:33229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 22:36:42 dist-test-slave-77v9 krb5kdc[342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401402, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.66@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:36:42.807003 1224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:42.791307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:59025 (local address 127.16.233.126:42443)
0810 22:36:42.791494 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 22:36:42.791499 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:36:42.792445 (+ 946us) server_negotiation.cc:407] Connection header received
0810 22:36:42.794048 (+ 1603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:36:42.794051 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:36:42.794136 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:36:42.794245 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:36:42.795697 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:42.796415 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:42.798405 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:42.798683 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:42.798779 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:42.802599 (+ 3820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:36:42.802603 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:36:42.802605 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:36:42.802638 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:36:42.804436 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:36:42.805185 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:36:42.805188 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:36:42.805190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:36:42.805277 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:36:42.805678 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:36:42.805682 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:36:42.805683 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:36:42.805909 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:36:42.806023 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 22:36:42.806632 (+ 609us) server_negotiation.cc:299] Negotiation successful
0810 22:36:42.806797 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260810 22:36:42.807930 1384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:42.791682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:42443 (local address 127.16.233.66:59025)
0810 22:36:42.792237 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:42.792309 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0810 22:36:42.793785 (+ 1476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:36:42.794550 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:36:42.794557 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:36:42.794962 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:36:42.795506 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:42.795521 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:42.796586 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:42.796591 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:42.798231 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:42.798249 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:42.798905 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:42.798911 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:42.799194 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:42.800101 (+ 907us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:36:42.800133 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:36:42.802310 (+ 2177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:36:42.804595 (+ 2285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:36:42.804602 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:36:42.804631 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:36:42.805047 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:36:42.805398 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:36:42.805403 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:36:42.805411 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:36:42.805567 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:36:42.806039 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:36:42.806048 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:36:42.806346 (+ 298us) client_negotiation.cc:769] Sending connection context
0810 22:36:42.806566 (+ 220us) client_negotiation.cc:240] Negotiation successful
0810 22:36:42.806967 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":130,"threads_started":1}
I20260810 22:36:42.809351 1382 heartbeater.cc:347] Connected to a master server at 127.16.233.126:42443
I20260810 22:36:42.809670 1382 heartbeater.cc:464] Registering TS with master...
I20260810 22:36:42.810402 1382 heartbeater.cc:511] Master 127.16.233.126:42443 requested a full tablet report, sending...
I20260810 22:36:42.811880 997 ts_manager.cc:194] Registered new tserver with Master: 10e4188e49c24a888f122a94d2349393 (127.16.233.66:41115)
I20260810 22:36:42.812798 997 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.66@KRBTEST.COM'} at 127.16.233.66:59025
W20260810 22:36:42.923446 1385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:36:42.923766 1385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:36:42.923815 1385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:36:42.928074 1385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:36:42.928171 1385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:36:42.928233 1385 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.67
I20260810 22:36:42.934006 1385 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:33229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab
--principal=kudu/127.16.233.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.67:46119
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.233.67
--webserver_port=42629
--tserver_master_addrs=127.16.233.126:42443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1385
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.67
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:36:42.935194 1385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:36:42.936234 1385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:36:42.946980 1391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:36:42.946975 1395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:36:42.946973 1392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:36:42.947227 1385 server_base.cc:1034] running on GCE node
I20260810 22:36:42.947934 1385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:36:42.948694 1385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:36:42.949924 1385 hybrid_clock.cc:648] HybridClock initialized: now 1786401402949904 us; error 38 us; skew 500 ppm
Aug 10 22:36:42 dist-test-slave-77v9 krb5kdc[342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401402, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:36:42.953316 1385 init.cc:377] Logged in from keytab as kudu/127.16.233.67@KRBTEST.COM (short username kudu)
I20260810 22:36:42.954418 1385 webserver.cc:460] Webserver started at http://127.16.233.67:42629/ using document root <none> and password file <none>
I20260810 22:36:42.955152 1385 fs_manager.cc:359] Metadata directory not provided
I20260810 22:36:42.955276 1385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:36:42.960031 1385 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 22:36:42.962760 1402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:36:42.964254 1385 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 22:36:42.964435 1385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-2/wal
uuid: "89ffcbdb9f8c45c7a748bca58e869e23"
format_stamp: "Formatted at 2026-08-10 22:36:41 on dist-test-slave-77v9"
server_key: "49282af95a15dc914a459555823d5293"
server_key_iv: "0d44a7994ff90ca1fff763e838475662"
server_key_version: "encryptionkey@0"
I20260810 22:36:42.965155 1385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:36:42.977542 1385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:36:42.981393 1385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:36:42.981632 1385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:36:42.982311 1385 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:36:42.983412 1385 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:36:42.983489 1385 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:36:42.983559 1385 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:36:42.983610 1385 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:36:42.994019 1385 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.67:46119
I20260810 22:36:42.994052 1535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.67:46119 every 8 connection(s)
I20260810 22:36:42.995204 1385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
I20260810 22:36:43.003974 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 1385
Aug 10 22:36:43 dist-test-slave-77v9 krb5kdc[342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401402, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.67@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:36:43.011739 1224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:42.997351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:45211 (local address 127.16.233.126:42443)
0810 22:36:42.997521 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 22:36:42.997526 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:36:42.998448 (+ 922us) server_negotiation.cc:407] Connection header received
0810 22:36:42.999971 (+ 1523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:36:42.999975 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:36:43.000040 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:36:43.000152 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:36:43.001636 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:43.002170 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:43.003378 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:43.003659 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:43.003779 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:43.006814 (+ 3035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:36:43.006817 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:36:43.006820 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:36:43.006852 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:36:43.009095 (+ 2243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:36:43.009789 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:36:43.009792 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:36:43.009793 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:36:43.009872 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:36:43.010305 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:36:43.010308 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:36:43.010309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:36:43.010558 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:36:43.010639 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 22:36:43.011379 (+ 740us) server_negotiation.cc:299] Negotiation successful
0810 22:36:43.011538 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 22:36:43.012676 1539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:42.997748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:42443 (local address 127.16.233.67:45211)
0810 22:36:42.998250 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:42.998299 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 22:36:42.999721 (+ 1422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:36:43.000349 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:36:43.000356 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:36:43.000746 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:36:43.001464 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:43.001480 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:43.002310 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:43.002314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:43.003236 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:43.003245 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:43.003842 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:43.003845 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:43.004032 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:43.004804 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:36:43.004854 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:36:43.006635 (+ 1781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:36:43.009234 (+ 2599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:36:43.009244 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:36:43.009266 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:36:43.009678 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:36:43.009977 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:36:43.009982 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:36:43.009985 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:36:43.010198 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:36:43.010673 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:36:43.010683 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:36:43.011049 (+ 366us) client_negotiation.cc:769] Sending connection context
0810 22:36:43.011307 (+ 258us) client_negotiation.cc:240] Negotiation successful
0810 22:36:43.011651 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":126,"threads_started":1}
I20260810 22:36:43.014037 1537 heartbeater.cc:347] Connected to a master server at 127.16.233.126:42443
I20260810 22:36:43.014179 1541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:43.004810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38940 (local address 127.16.233.126:42443)
0810 22:36:43.005146 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0810 22:36:43.005151 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:36:43.005166 (+ 15us) server_negotiation.cc:407] Connection header received
0810 22:36:43.005334 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:36:43.005337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:36:43.005390 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:36:43.005494 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:36:43.006238 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:43.006859 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:43.008019 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:43.008394 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:43.008504 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:43.010211 (+ 1707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:36:43.010214 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:36:43.010216 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:36:43.010235 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:36:43.012027 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:36:43.012553 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:36:43.012559 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:36:43.012561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:36:43.012645 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:36:43.012993 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:36:43.012997 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:36:43.012998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:36:43.013242 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:36:43.013454 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0810 22:36:43.013856 (+ 402us) server_negotiation.cc:299] Negotiation successful
0810 22:36:43.013994 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":127,"threads_started":1}
I20260810 22:36:43.014333 1537 heartbeater.cc:464] Registering TS with master...
I20260810 22:36:43.014909 1537 heartbeater.cc:511] Master 127.16.233.126:42443 requested a full tablet report, sending...
I20260810 22:36:43.016115 996 ts_manager.cc:194] Registered new tserver with Master: 89ffcbdb9f8c45c7a748bca58e869e23 (127.16.233.67:46119)
I20260810 22:36:43.016896 996 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.67@KRBTEST.COM'} at 127.16.233.67:45211
I20260810 22:36:43.017695 17317 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 22:36:43.022207 1541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:43.018687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38950 (local address 127.16.233.126:42443)
0810 22:36:43.018847 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 22:36:43.018851 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:36:43.018866 (+ 15us) server_negotiation.cc:407] Connection header received
0810 22:36:43.018952 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:36:43.018954 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:36:43.019067 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:36:43.019153 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:36:43.019366 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:43.019866 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:43.020725 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:43.021051 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:43.021138 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:43.021437 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:36:43.021847 (+ 410us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:36:43.021943 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 22:36:43.022023 (+ 80us) server_negotiation.cc:299] Negotiation successful
0810 22:36:43.022084 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 22:36:43.023028 997 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:38950:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 22:36:43.026914 997 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:36:43.036098 1142 tablet_service.cc:1467] Processing CreateTablet for tablet 5a1826d942b64ea3ab6cd117740f3d7e (DEFAULT_TABLE table=test-table [id=cadea2971d76414eb21a6a8a262183ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:36:43.036902 1142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a1826d942b64ea3ab6cd117740f3d7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:36:43.039953 1092 tablet_bootstrap.cc:492] T 5a1826d942b64ea3ab6cd117740f3d7e P 7285c96084ea4dd5a35d59d5b2b47537: Bootstrap starting.
I20260810 22:36:43.041229 1092 tablet_bootstrap.cc:654] T 5a1826d942b64ea3ab6cd117740f3d7e P 7285c96084ea4dd5a35d59d5b2b47537: Neither blocks nor log segments found. Creating new log.
I20260810 22:36:43.041247 1229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:43.034952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.66:41115 (local address 127.0.0.1:33158)
0810 22:36:43.035293 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:43.035311 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 22:36:43.035399 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:36:43.036526 (+ 1127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:36:43.036529 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:36:43.036550 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:36:43.036659 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:43.036666 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:43.038167 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:43.038170 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:43.039382 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:43.039397 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:43.040730 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:43.040736 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:43.040870 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:43.040901 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 22:36:43.040996 (+ 95us) client_negotiation.cc:240] Negotiation successful
0810 22:36:43.041094 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261}
I20260810 22:36:43.041407 1549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:43.035188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33158 (local address 127.16.233.66:41115)
0810 22:36:43.036001 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0810 22:36:43.036012 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 22:36:43.036040 (+ 28us) server_negotiation.cc:407] Connection header received
0810 22:36:43.036115 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:36:43.036119 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:36:43.036309 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:36:43.036525 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:36:43.036804 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:43.037967 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:43.039576 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:43.040579 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:43.040714 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:43.040918 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0810 22:36:43.041068 (+ 150us) server_negotiation.cc:299] Negotiation successful
0810 22:36:43.041223 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":712,"thread_start_us":169,"threads_started":1}
I20260810 22:36:43.042761 1092 tablet_bootstrap.cc:492] T 5a1826d942b64ea3ab6cd117740f3d7e P 7285c96084ea4dd5a35d59d5b2b47537: No bootstrap required, opened a new log
I20260810 22:36:43.042878 1092 ts_tablet_manager.cc:1397] T 5a1826d942b64ea3ab6cd117740f3d7e P 7285c96084ea4dd5a35d59d5b2b47537: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 22:36:43.043190 1547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:43.034992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.67:46119 (local address 127.0.0.1:52506)
0810 22:36:43.035503 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:43.035522 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 22:36:43.035629 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:36:43.037445 (+ 1816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:36:43.037448 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:36:43.037471 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:36:43.037590 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:43.037597 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:43.039537 (+ 1940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:43.039544 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:43.040594 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:43.040605 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:43.042790 (+ 2185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:43.042794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:43.042898 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:43.042936 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 22:36:43.042994 (+ 58us) client_negotiation.cc:240] Negotiation successful
0810 22:36:43.043074 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":154,"threads_started":1}
I20260810 22:36:43.043362 1092 raft_consensus.cc:348] T 5a1826d942b64ea3ab6cd117740f3d7e P 7285c96084ea4dd5a35d59d5b2b47537 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10e4188e49c24a888f122a94d2349393" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 41115 } } peers { permanent_uuid: "89ffcbdb9f8c45c7a748bca58e869e23" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 46119 } } peers { permanent_uuid: "7285c96084ea4dd5a35d59d5b2b47537" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 41675 } }
I20260810 22:36:43.043529 1092 raft_consensus.cc:374] T 5a1826d942b64ea3ab6cd117740f3d7e P 7285c96084ea4dd5a35d59d5b2b47537 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:36:43.043586 1092 raft_consensus.cc:729] T 5a1826d942b64ea3ab6cd117740f3d7e P 7285c96084ea4dd5a35d59d5b2b47537 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7285c96084ea4dd5a35d59d5b2b47537, State: Initialized, Role: FOLLOWER
I20260810 22:36:43.043749 1092 consensus_queue.cc:260] T 5a1826d942b64ea3ab6cd117740f3d7e P 7285c96084ea4dd5a35d59d5b2b47537 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10e4188e49c24a888f122a94d2349393" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 41115 } } peers { permanent_uuid: "89ffcbdb9f8c45c7a748bca58e869e23" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 46119 } } peers { permanent_uuid: "7285c96084ea4dd5a35d59d5b2b47537" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 41675 } }
I20260810 22:36:43.044097 1550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:43.035411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52506 (local address 127.16.233.67:46119)
0810 22:36:43.036927 (+ 1516us) server_negotiation.cc:206] Beginning negotiation
0810 22:36:43.036942 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 22:36:43.036968 (+ 26us) server_negotiation.cc:407] Connection header received
0810 22:36:43.037055 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:36:43.037060 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:36:43.037255 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:36:43.037627 (+ 372us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:36:43.038010 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:43.039362 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:43.040773 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:43.042646 (+ 1873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:43.043545 (+ 899us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:43.043730 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0810 22:36:43.043799 (+ 69us) server_negotiation.cc:299] Negotiation successful
0810 22:36:43.043927 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1370,"thread_start_us":117,"threads_started":1}
I20260810 22:36:43.044166 1092 ts_tablet_manager.cc:1428] T 5a1826d942b64ea3ab6cd117740f3d7e P 7285c96084ea4dd5a35d59d5b2b47537: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260810 22:36:43.045094 1300 tablet_service.cc:1467] Processing CreateTablet for tablet 5a1826d942b64ea3ab6cd117740f3d7e (DEFAULT_TABLE table=test-table [id=cadea2971d76414eb21a6a8a262183ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:36:43.046564 1300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a1826d942b64ea3ab6cd117740f3d7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:36:43.047370 1459 tablet_service.cc:1467] Processing CreateTablet for tablet 5a1826d942b64ea3ab6cd117740f3d7e (DEFAULT_TABLE table=test-table [id=cadea2971d76414eb21a6a8a262183ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:36:43.048571 1459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a1826d942b64ea3ab6cd117740f3d7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:36:43.056246 1552 tablet_bootstrap.cc:492] T 5a1826d942b64ea3ab6cd117740f3d7e P 10e4188e49c24a888f122a94d2349393: Bootstrap starting.
I20260810 22:36:43.057502 1553 tablet_bootstrap.cc:492] T 5a1826d942b64ea3ab6cd117740f3d7e P 89ffcbdb9f8c45c7a748bca58e869e23: Bootstrap starting.
I20260810 22:36:43.059523 1552 tablet_bootstrap.cc:654] T 5a1826d942b64ea3ab6cd117740f3d7e P 10e4188e49c24a888f122a94d2349393: Neither blocks nor log segments found. Creating new log.
I20260810 22:36:43.059825 1553 tablet_bootstrap.cc:654] T 5a1826d942b64ea3ab6cd117740f3d7e P 89ffcbdb9f8c45c7a748bca58e869e23: Neither blocks nor log segments found. Creating new log.
I20260810 22:36:43.060580 1552 log.cc:824] T 5a1826d942b64ea3ab6cd117740f3d7e P 10e4188e49c24a888f122a94d2349393: Log is configured to *not* fsync() on all Append() calls
I20260810 22:36:43.061141 1553 log.cc:824] T 5a1826d942b64ea3ab6cd117740f3d7e P 89ffcbdb9f8c45c7a748bca58e869e23: Log is configured to *not* fsync() on all Append() calls
I20260810 22:36:43.063067 1552 tablet_bootstrap.cc:492] T 5a1826d942b64ea3ab6cd117740f3d7e P 10e4188e49c24a888f122a94d2349393: No bootstrap required, opened a new log
W20260810 22:36:43.063220 1223 tablet.cc:2369] T 5a1826d942b64ea3ab6cd117740f3d7e P 7285c96084ea4dd5a35d59d5b2b47537: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:36:43.063330 1552 ts_tablet_manager.cc:1397] T 5a1826d942b64ea3ab6cd117740f3d7e P 10e4188e49c24a888f122a94d2349393: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 22:36:43.063493 1553 tablet_bootstrap.cc:492] T 5a1826d942b64ea3ab6cd117740f3d7e P 89ffcbdb9f8c45c7a748bca58e869e23: No bootstrap required, opened a new log
I20260810 22:36:43.063690 1553 ts_tablet_manager.cc:1397] T 5a1826d942b64ea3ab6cd117740f3d7e P 89ffcbdb9f8c45c7a748bca58e869e23: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 22:36:43.067085 1552 raft_consensus.cc:348] T 5a1826d942b64ea3ab6cd117740f3d7e P 10e4188e49c24a888f122a94d2349393 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10e4188e49c24a888f122a94d2349393" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 41115 } } peers { permanent_uuid: "89ffcbdb9f8c45c7a748bca58e869e23" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 46119 } } peers { permanent_uuid: "7285c96084ea4dd5a35d59d5b2b47537" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 41675 } }
I20260810 22:36:43.067094 1553 raft_consensus.cc:348] T 5a1826d942b64ea3ab6cd117740f3d7e P 89ffcbdb9f8c45c7a748bca58e869e23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10e4188e49c24a888f122a94d2349393" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 41115 } } peers { permanent_uuid: "89ffcbdb9f8c45c7a748bca58e869e23" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 46119 } } peers { permanent_uuid: "7285c96084ea4dd5a35d59d5b2b47537" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 41675 } }
I20260810 22:36:43.067399 1553 raft_consensus.cc:374] T 5a1826d942b64ea3ab6cd117740f3d7e P 89ffcbdb9f8c45c7a748bca58e869e23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:36:43.067404 1552 raft_consensus.cc:374] T 5a1826d942b64ea3ab6cd117740f3d7e P 10e4188e49c24a888f122a94d2349393 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:36:43.067502 1553 raft_consensus.cc:729] T 5a1826d942b64ea3ab6cd117740f3d7e P 89ffcbdb9f8c45c7a748bca58e869e23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89ffcbdb9f8c45c7a748bca58e869e23, State: Initialized, Role: FOLLOWER
I20260810 22:36:43.067512 1552 raft_consensus.cc:729] T 5a1826d942b64ea3ab6cd117740f3d7e P 10e4188e49c24a888f122a94d2349393 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10e4188e49c24a888f122a94d2349393, State: Initialized, Role: FOLLOWER
I20260810 22:36:43.068027 1552 consensus_queue.cc:260] T 5a1826d942b64ea3ab6cd117740f3d7e P 10e4188e49c24a888f122a94d2349393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10e4188e49c24a888f122a94d2349393" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 41115 } } peers { permanent_uuid: "89ffcbdb9f8c45c7a748bca58e869e23" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 46119 } } peers { permanent_uuid: "7285c96084ea4dd5a35d59d5b2b47537" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 41675 } }
I20260810 22:36:43.068027 1553 consensus_queue.cc:260] T 5a1826d942b64ea3ab6cd117740f3d7e P 89ffcbdb9f8c45c7a748bca58e869e23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10e4188e49c24a888f122a94d2349393" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 41115 } } peers { permanent_uuid: "89ffcbdb9f8c45c7a748bca58e869e23" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 46119 } } peers { permanent_uuid: "7285c96084ea4dd5a35d59d5b2b47537" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 41675 } }
I20260810 22:36:43.068921 1382 heartbeater.cc:503] Master 127.16.233.126:42443 was elected leader, sending a full tablet report...
I20260810 22:36:43.069409 1553 ts_tablet_manager.cc:1428] T 5a1826d942b64ea3ab6cd117740f3d7e P 89ffcbdb9f8c45c7a748bca58e869e23: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 22:36:43.069501 1537 heartbeater.cc:503] Master 127.16.233.126:42443 was elected leader, sending a full tablet report...
I20260810 22:36:43.070475 1552 ts_tablet_manager.cc:1428] T 5a1826d942b64ea3ab6cd117740f3d7e P 10e4188e49c24a888f122a94d2349393: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260810 22:36:43.246395 1538 tablet.cc:2369] T 5a1826d942b64ea3ab6cd117740f3d7e P 89ffcbdb9f8c45c7a748bca58e869e23: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 22:36:43.290459 1383 tablet.cc:2369] T 5a1826d942b64ea3ab6cd117740f3d7e P 10e4188e49c24a888f122a94d2349393: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:36:43.378675 1557 raft_consensus.cc:482] T 5a1826d942b64ea3ab6cd117740f3d7e P 89ffcbdb9f8c45c7a748bca58e869e23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:36:43.378990 1557 raft_consensus.cc:504] T 5a1826d942b64ea3ab6cd117740f3d7e P 89ffcbdb9f8c45c7a748bca58e869e23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10e4188e49c24a888f122a94d2349393" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 41115 } } peers { permanent_uuid: "89ffcbdb9f8c45c7a748bca58e869e23" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 46119 } } peers { permanent_uuid: "7285c96084ea4dd5a35d59d5b2b47537" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 41675 } }
I20260810 22:36:43.380731 1557 leader_election.cc:288] T 5a1826d942b64ea3ab6cd117740f3d7e P 89ffcbdb9f8c45c7a748bca58e869e23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10e4188e49c24a888f122a94d2349393 (127.16.233.66:41115), 7285c96084ea4dd5a35d59d5b2b47537 (127.16.233.65:41675)
I20260810 22:36:43.386145 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:43.381525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:58793 (local address 127.16.233.65:41675)
0810 22:36:43.382107 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0810 22:36:43.382113 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:36:43.382131 (+ 18us) server_negotiation.cc:407] Connection header received
0810 22:36:43.382201 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:36:43.382204 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:36:43.382262 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:36:43.382574 (+ 312us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:36:43.382820 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:43.383651 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:43.385009 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:43.385469 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:43.385627 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:43.385689 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 22:36:43.385889 (+ 200us) server_negotiation.cc:299] Negotiation successful
0810 22:36:43.385971 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":127,"threads_started":1}
I20260810 22:36:43.386145 1539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:43.381081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.65:41675 (local address 127.16.233.67:58793)
0810 22:36:43.381271 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:43.381289 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 22:36:43.381415 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:36:43.382440 (+ 1025us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:36:43.382444 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:36:43.382496 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:36:43.382664 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:43.382700 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:43.383795 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:43.383802 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:43.384801 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:43.384814 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:43.385627 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:43.385631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:43.385745 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:43.385767 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 22:36:43.385868 (+ 101us) client_negotiation.cc:240] Negotiation successful
0810 22:36:43.385966 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"mutex_wait_us":25,"spinlock_wait_cycles":6912}
I20260810 22:36:43.386200 1549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:43.381180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:35849 (local address 127.16.233.66:41115)
0810 22:36:43.381327 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 22:36:43.381332 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:36:43.381974 (+ 642us) server_negotiation.cc:407] Connection header received
0810 22:36:43.382139 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:36:43.382143 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:36:43.382256 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:36:43.382349 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:36:43.382771 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:43.383521 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:43.385075 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:43.385547 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:43.385707 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:43.385764 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 22:36:43.385973 (+ 209us) server_negotiation.cc:299] Negotiation successful
0810 22:36:43.386056 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 22:36:43.386301 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:43.381426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.66:41115 (local address 127.16.233.67:35849)
0810 22:36:43.381860 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:43.381873 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 22:36:43.382010 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:36:43.382377 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:36:43.382380 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:36:43.382397 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:36:43.382638 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:43.382649 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:43.383705 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:43.383708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:43.384921 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:43.384934 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:43.385699 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:43.385704 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:43.385819 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:43.385843 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 22:36:43.385954 (+ 111us) client_negotiation.cc:240] Negotiation successful
0810 22:36:43.386036 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":49,"spinlock_wait_cycles":69504,"thread_start_us":175,"threads_started":1}
I20260810 22:36:43.387104 1329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a1826d942b64ea3ab6cd117740f3d7e" candidate_uuid: "89ffcbdb9f8c45c7a748bca58e869e23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10e4188e49c24a888f122a94d2349393" is_pre_election: true
I20260810 22:36:43.387283 1166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a1826d942b64ea3ab6cd117740f3d7e" candidate_uuid: "89ffcbdb9f8c45c7a748bca58e869e23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7285c96084ea4dd5a35d59d5b2b47537" is_pre_election: true
I20260810 22:36:43.387576 1166 raft_consensus.cc:2454] T 5a1826d942b64ea3ab6cd117740f3d7e P 7285c96084ea4dd5a35d59d5b2b47537 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89ffcbdb9f8c45c7a748bca58e869e23 in term 0.
I20260810 22:36:43.387537 1329 raft_consensus.cc:2454] T 5a1826d942b64ea3ab6cd117740f3d7e P 10e4188e49c24a888f122a94d2349393 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89ffcbdb9f8c45c7a748bca58e869e23 in term 0.
I20260810 22:36:43.388144 1403 leader_election.cc:302] T 5a1826d942b64ea3ab6cd117740f3d7e P 89ffcbdb9f8c45c7a748bca58e869e23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10e4188e49c24a888f122a94d2349393, 89ffcbdb9f8c45c7a748bca58e869e23; no voters:
I20260810 22:36:43.388530 1557 raft_consensus.cc:2781] T 5a1826d942b64ea3ab6cd117740f3d7e P 89ffcbdb9f8c45c7a748bca58e869e23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:36:43.388641 1557 raft_consensus.cc:482] T 5a1826d942b64ea3ab6cd117740f3d7e P 89ffcbdb9f8c45c7a748bca58e869e23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:36:43.388705 1557 raft_consensus.cc:3037] T 5a1826d942b64ea3ab6cd117740f3d7e P 89ffcbdb9f8c45c7a748bca58e869e23 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:36:43.389910 1557 raft_consensus.cc:504] T 5a1826d942b64ea3ab6cd117740f3d7e P 89ffcbdb9f8c45c7a748bca58e869e23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10e4188e49c24a888f122a94d2349393" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 41115 } } peers { permanent_uuid: "89ffcbdb9f8c45c7a748bca58e869e23" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 46119 } } peers { permanent_uuid: "7285c96084ea4dd5a35d59d5b2b47537" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 41675 } }
I20260810 22:36:43.390326 1557 leader_election.cc:288] T 5a1826d942b64ea3ab6cd117740f3d7e P 89ffcbdb9f8c45c7a748bca58e869e23 [CANDIDATE]: Term 1 election: Requested vote from peers 10e4188e49c24a888f122a94d2349393 (127.16.233.66:41115), 7285c96084ea4dd5a35d59d5b2b47537 (127.16.233.65:41675)
I20260810 22:36:43.390846 1329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a1826d942b64ea3ab6cd117740f3d7e" candidate_uuid: "89ffcbdb9f8c45c7a748bca58e869e23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10e4188e49c24a888f122a94d2349393"
I20260810 22:36:43.391070 1166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a1826d942b64ea3ab6cd117740f3d7e" candidate_uuid: "89ffcbdb9f8c45c7a748bca58e869e23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7285c96084ea4dd5a35d59d5b2b47537"
I20260810 22:36:43.391069 1329 raft_consensus.cc:3037] T 5a1826d942b64ea3ab6cd117740f3d7e P 10e4188e49c24a888f122a94d2349393 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:36:43.391238 1166 raft_consensus.cc:3037] T 5a1826d942b64ea3ab6cd117740f3d7e P 7285c96084ea4dd5a35d59d5b2b47537 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:36:43.392689 1329 raft_consensus.cc:2454] T 5a1826d942b64ea3ab6cd117740f3d7e P 10e4188e49c24a888f122a94d2349393 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89ffcbdb9f8c45c7a748bca58e869e23 in term 1.
I20260810 22:36:43.392689 1166 raft_consensus.cc:2454] T 5a1826d942b64ea3ab6cd117740f3d7e P 7285c96084ea4dd5a35d59d5b2b47537 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89ffcbdb9f8c45c7a748bca58e869e23 in term 1.
I20260810 22:36:43.393182 1403 leader_election.cc:302] T 5a1826d942b64ea3ab6cd117740f3d7e P 89ffcbdb9f8c45c7a748bca58e869e23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10e4188e49c24a888f122a94d2349393, 89ffcbdb9f8c45c7a748bca58e869e23; no voters:
I20260810 22:36:43.393450 1557 raft_consensus.cc:2781] T 5a1826d942b64ea3ab6cd117740f3d7e P 89ffcbdb9f8c45c7a748bca58e869e23 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:36:43.393720 1557 raft_consensus.cc:686] T 5a1826d942b64ea3ab6cd117740f3d7e P 89ffcbdb9f8c45c7a748bca58e869e23 [term 1 LEADER]: Becoming Leader. State: Replica: 89ffcbdb9f8c45c7a748bca58e869e23, State: Running, Role: LEADER
I20260810 22:36:43.394146 1557 consensus_queue.cc:237] T 5a1826d942b64ea3ab6cd117740f3d7e P 89ffcbdb9f8c45c7a748bca58e869e23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10e4188e49c24a888f122a94d2349393" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 41115 } } peers { permanent_uuid: "89ffcbdb9f8c45c7a748bca58e869e23" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 46119 } } peers { permanent_uuid: "7285c96084ea4dd5a35d59d5b2b47537" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 41675 } }
I20260810 22:36:43.397660 996 catalog_manager.cc:5495] T 5a1826d942b64ea3ab6cd117740f3d7e P 89ffcbdb9f8c45c7a748bca58e869e23 reported cstate change: term changed from 0 to 1, leader changed from <none> to 89ffcbdb9f8c45c7a748bca58e869e23 (127.16.233.67). New cstate: current_term: 1 leader_uuid: "89ffcbdb9f8c45c7a748bca58e869e23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10e4188e49c24a888f122a94d2349393" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 41115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89ffcbdb9f8c45c7a748bca58e869e23" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 46119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7285c96084ea4dd5a35d59d5b2b47537" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 41675 } health_report { overall_health: UNKNOWN } } }
I20260810 22:36:43.415493 1550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:43.411498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52520 (local address 127.16.233.67:46119)
0810 22:36:43.411635 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 22:36:43.411639 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:36:43.411654 (+ 15us) server_negotiation.cc:407] Connection header received
0810 22:36:43.411741 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:36:43.411744 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:36:43.411804 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:36:43.411903 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:36:43.412137 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:43.412865 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:43.413853 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:43.414097 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:43.414193 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:43.414493 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:36:43.415013 (+ 520us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:36:43.415116 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 22:36:43.415295 (+ 179us) server_negotiation.cc:299] Negotiation successful
0810 22:36:43.415357 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 22:36:43.422958 1166 raft_consensus.cc:1261] T 5a1826d942b64ea3ab6cd117740f3d7e P 7285c96084ea4dd5a35d59d5b2b47537 [term 1 FOLLOWER]: Refusing update from remote peer 89ffcbdb9f8c45c7a748bca58e869e23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:36:43.422961 1329 raft_consensus.cc:1261] T 5a1826d942b64ea3ab6cd117740f3d7e P 10e4188e49c24a888f122a94d2349393 [term 1 FOLLOWER]: Refusing update from remote peer 89ffcbdb9f8c45c7a748bca58e869e23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:36:43.423795 1560 consensus_queue.cc:1035] T 5a1826d942b64ea3ab6cd117740f3d7e P 89ffcbdb9f8c45c7a748bca58e869e23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7285c96084ea4dd5a35d59d5b2b47537" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 41675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:36:43.424027 1557 consensus_queue.cc:1035] T 5a1826d942b64ea3ab6cd117740f3d7e P 89ffcbdb9f8c45c7a748bca58e869e23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10e4188e49c24a888f122a94d2349393" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 41115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:36:43.434113 1565 mvcc.cc:205] Tried to move back new op lower bound from 7317100148412137472 to 7317100148305293312. Current Snapshot: MvccSnapshot[applied={T|T < 7317100148412137472}]
I20260810 22:36:43.435843 1563 mvcc.cc:205] Tried to move back new op lower bound from 7317100148412137472 to 7317100148305293312. Current Snapshot: MvccSnapshot[applied={T|T < 7317100148412137472 or (T in {7317100148412137472})}]
I20260810 22:36:43.443735 997 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:38950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:36:43.444001 997 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:38950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:36:43.446864 997 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 84f10ccd7de840299a6c6b5a936dc470: Sending DeleteTablet for 3 replicas of tablet 5a1826d942b64ea3ab6cd117740f3d7e
I20260810 22:36:43.447685 1142 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a1826d942b64ea3ab6cd117740f3d7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:36:43 UTC) from {username='kudu', principal='kudu/127.16.233.126@KRBTEST.COM'} at 127.0.0.1:35272
I20260810 22:36:43.447759 1459 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a1826d942b64ea3ab6cd117740f3d7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:36:43 UTC) from {username='kudu', principal='kudu/127.16.233.126@KRBTEST.COM'} at 127.0.0.1:52506
I20260810 22:36:43.448395 1575 tablet_replica.cc:330] T 5a1826d942b64ea3ab6cd117740f3d7e P 7285c96084ea4dd5a35d59d5b2b47537: stopping tablet replica
I20260810 22:36:43.448339 1300 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a1826d942b64ea3ab6cd117740f3d7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:36:43 UTC) from {username='kudu', principal='kudu/127.16.233.126@KRBTEST.COM'} at 127.0.0.1:33158
I20260810 22:36:43.448539 1575 raft_consensus.cc:2229] T 5a1826d942b64ea3ab6cd117740f3d7e P 7285c96084ea4dd5a35d59d5b2b47537 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:36:43.448706 1575 raft_consensus.cc:2258] T 5a1826d942b64ea3ab6cd117740f3d7e P 7285c96084ea4dd5a35d59d5b2b47537 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:36:43.448745 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 1040
I20260810 22:36:43.450953 1577 tablet_replica.cc:330] T 5a1826d942b64ea3ab6cd117740f3d7e P 10e4188e49c24a888f122a94d2349393: stopping tablet replica
I20260810 22:36:43.451388 1577 raft_consensus.cc:2229] T 5a1826d942b64ea3ab6cd117740f3d7e P 10e4188e49c24a888f122a94d2349393 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:36:43.451735 1577 raft_consensus.cc:2258] T 5a1826d942b64ea3ab6cd117740f3d7e P 10e4188e49c24a888f122a94d2349393 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:36:43.452030 1576 tablet_replica.cc:330] T 5a1826d942b64ea3ab6cd117740f3d7e P 89ffcbdb9f8c45c7a748bca58e869e23: stopping tablet replica
I20260810 22:36:43.452360 1576 raft_consensus.cc:2229] T 5a1826d942b64ea3ab6cd117740f3d7e P 89ffcbdb9f8c45c7a748bca58e869e23 [term 1 LEADER]: Raft consensus shutting down.
I20260810 22:36:43.452736 1576 raft_consensus.cc:2258] T 5a1826d942b64ea3ab6cd117740f3d7e P 89ffcbdb9f8c45c7a748bca58e869e23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:36:43.454810 1576 ts_tablet_manager.cc:1905] T 5a1826d942b64ea3ab6cd117740f3d7e P 89ffcbdb9f8c45c7a748bca58e869e23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:36:43.458954 1576 ts_tablet_manager.cc:1918] T 5a1826d942b64ea3ab6cd117740f3d7e P 89ffcbdb9f8c45c7a748bca58e869e23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:36:43.459141 1576 log.cc:1196] T 5a1826d942b64ea3ab6cd117740f3d7e P 89ffcbdb9f8c45c7a748bca58e869e23: Deleting WAL directory at /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-2/wal/wals/5a1826d942b64ea3ab6cd117740f3d7e
I20260810 22:36:43.459709 1576 ts_tablet_manager.cc:1939] T 5a1826d942b64ea3ab6cd117740f3d7e P 89ffcbdb9f8c45c7a748bca58e869e23: Deleting consensus metadata
I20260810 22:36:43.461643 983 catalog_manager.cc:4841] TS 89ffcbdb9f8c45c7a748bca58e869e23 (127.16.233.67:46119): tablet 5a1826d942b64ea3ab6cd117740f3d7e (table test-table [id=cadea2971d76414eb21a6a8a262183ef]) successfully deleted
I20260810 22:36:43.468008 1577 ts_tablet_manager.cc:1905] T 5a1826d942b64ea3ab6cd117740f3d7e P 10e4188e49c24a888f122a94d2349393: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:36:43.471231 1577 ts_tablet_manager.cc:1918] T 5a1826d942b64ea3ab6cd117740f3d7e P 10e4188e49c24a888f122a94d2349393: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:36:43.471359 1577 log.cc:1196] T 5a1826d942b64ea3ab6cd117740f3d7e P 10e4188e49c24a888f122a94d2349393: Deleting WAL directory at /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-1/wal/wals/5a1826d942b64ea3ab6cd117740f3d7e
I20260810 22:36:43.471705 1577 ts_tablet_manager.cc:1939] T 5a1826d942b64ea3ab6cd117740f3d7e P 10e4188e49c24a888f122a94d2349393: Deleting consensus metadata
I20260810 22:36:43.472425 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 1227
W20260810 22:36:43.472474 981 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.233.65:41675 (error 108)
I20260810 22:36:43.475777 1547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:43.475373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.65:41675 (local address 127.0.0.1:35284)
0810 22:36:43.475586 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:43.475674 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.65:41675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
W20260810 22:36:43.476032 981 catalog_manager.cc:4584] TS 7285c96084ea4dd5a35d59d5b2b47537 (127.16.233.65:41675): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5a1826d942b64ea3ab6cd117740f3d7e: Network error: Client connection negotiation failed: client connection to 127.16.233.65:41675: connect: Connection refused (error 111)
I20260810 22:36:43.481619 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 1385
I20260810 22:36:43.482890 1547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:43.481914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.66:41115 (local address 127.0.0.1:33168)
0810 22:36:43.482725 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:43.482795 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.66:41115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":710}
I20260810 22:36:43.490584 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 960
I20260810 22:36:43.499907 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:42443
--webserver_interface=127.16.233.126
--webserver_port=32945
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab
--principal=kudu/127.16.233.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:33229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.126:42443
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:36:43.621417 1579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:36:43.621807 1579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:36:43.621902 1579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:36:43.626542 1579 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:36:43.626664 1579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:36:43.626699 1579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:36:43.626736 1579 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:36:43.626775 1579 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:36:43.632859 1579 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:33229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.126:42443
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab
--principal=kudu/127.16.233.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:42443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.126
--webserver_port=32945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:36:43.634176 1579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:36:43.635317 1579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:36:43.644887 1587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:36:43.644891 1585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:36:43.645196 1589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:36:43.645004 1579 server_base.cc:1034] running on GCE node
I20260810 22:36:43.645953 1579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:36:43.647465 1579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:36:43.648721 1579 hybrid_clock.cc:648] HybridClock initialized: now 1786401403648689 us; error 52 us; skew 500 ppm
Aug 10 22:36:43 dist-test-slave-77v9 krb5kdc[342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401403, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:36:43.652372 1579 init.cc:377] Logged in from keytab as kudu/127.16.233.126@KRBTEST.COM (short username kudu)
I20260810 22:36:43.653432 1579 webserver.cc:460] Webserver started at http://127.16.233.126:32945/ using document root <none> and password file <none>
I20260810 22:36:43.654023 1579 fs_manager.cc:359] Metadata directory not provided
I20260810 22:36:43.654099 1579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:36:43.658567 1579 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 22:36:43.661283 1597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:36:43.662883 1579 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 22:36:43.663021 1579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/master-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/master-0/wal
uuid: "84f10ccd7de840299a6c6b5a936dc470"
format_stamp: "Formatted at 2026-08-10 22:36:40 on dist-test-slave-77v9"
server_key: "b530c58bf23da5e94b75c574b67feeee"
server_key_iv: "9e811611ea6a5088cd208ce6aed8b595"
server_key_version: "encryptionkey@0"
I20260810 22:36:43.663640 1579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:36:43.686966 1579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:36:43.690865 1579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:36:43.691048 1579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:36:43.699676 1579 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.126:42443
I20260810 22:36:43.699702 1654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.126:42443 every 8 connection(s)
I20260810 22:36:43.701049 1579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
I20260810 22:36:43.706267 1655 sys_catalog.cc:263] Verifying existing consensus state
I20260810 22:36:43.708416 1655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84f10ccd7de840299a6c6b5a936dc470: Bootstrap starting.
I20260810 22:36:43.709043 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 1579
I20260810 22:36:43.709575 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.65:41675
--local_ip_for_outbound_sockets=127.16.233.65
--tserver_master_addrs=127.16.233.126:42443
--webserver_port=37969
--webserver_interface=127.16.233.65
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab
--principal=kudu/127.16.233.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:33229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 22:36:43.730734 1655 log.cc:824] T 00000000000000000000000000000000 P 84f10ccd7de840299a6c6b5a936dc470: Log is configured to *not* fsync() on all Append() calls
I20260810 22:36:43.743326 1655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84f10ccd7de840299a6c6b5a936dc470: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 22:36:43.743844 1655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84f10ccd7de840299a6c6b5a936dc470: Bootstrap complete.
I20260810 22:36:43.748629 1655 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84f10ccd7de840299a6c6b5a936dc470 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84f10ccd7de840299a6c6b5a936dc470" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 42443 } }
I20260810 22:36:43.749279 1655 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84f10ccd7de840299a6c6b5a936dc470 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84f10ccd7de840299a6c6b5a936dc470, State: Initialized, Role: FOLLOWER
I20260810 22:36:43.749838 1655 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84f10ccd7de840299a6c6b5a936dc470 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84f10ccd7de840299a6c6b5a936dc470" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 42443 } }
I20260810 22:36:43.750015 1655 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84f10ccd7de840299a6c6b5a936dc470 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:36:43.750118 1655 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84f10ccd7de840299a6c6b5a936dc470 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:36:43.750253 1655 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84f10ccd7de840299a6c6b5a936dc470 [term 2 FOLLOWER]: Advancing to term 3
I20260810 22:36:43.752534 1655 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84f10ccd7de840299a6c6b5a936dc470 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84f10ccd7de840299a6c6b5a936dc470" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 42443 } }
I20260810 22:36:43.753015 1655 leader_election.cc:302] T 00000000000000000000000000000000 P 84f10ccd7de840299a6c6b5a936dc470 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84f10ccd7de840299a6c6b5a936dc470; no voters:
I20260810 22:36:43.753428 1655 leader_election.cc:288] T 00000000000000000000000000000000 P 84f10ccd7de840299a6c6b5a936dc470 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260810 22:36:43.753543 1659 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84f10ccd7de840299a6c6b5a936dc470 [term 3 FOLLOWER]: Leader election won for term 3
I20260810 22:36:43.753762 1659 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84f10ccd7de840299a6c6b5a936dc470 [term 3 LEADER]: Becoming Leader. State: Replica: 84f10ccd7de840299a6c6b5a936dc470, State: Running, Role: LEADER
I20260810 22:36:43.754173 1659 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84f10ccd7de840299a6c6b5a936dc470 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84f10ccd7de840299a6c6b5a936dc470" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 42443 } }
I20260810 22:36:43.754863 1655 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84f10ccd7de840299a6c6b5a936dc470 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:36:43.755797 1660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84f10ccd7de840299a6c6b5a936dc470 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "84f10ccd7de840299a6c6b5a936dc470" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84f10ccd7de840299a6c6b5a936dc470" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 42443 } } }
I20260810 22:36:43.755959 1660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84f10ccd7de840299a6c6b5a936dc470 [sys.catalog]: This master's current role is: LEADER
I20260810 22:36:43.756516 1661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84f10ccd7de840299a6c6b5a936dc470 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84f10ccd7de840299a6c6b5a936dc470. Latest consensus state: current_term: 3 leader_uuid: "84f10ccd7de840299a6c6b5a936dc470" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84f10ccd7de840299a6c6b5a936dc470" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 42443 } } }
I20260810 22:36:43.756690 1661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84f10ccd7de840299a6c6b5a936dc470 [sys.catalog]: This master's current role is: LEADER
I20260810 22:36:43.757074 1669 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:36:43.762831 1669 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:36:43.763759 1669 catalog_manager.cc:1259] Loaded cluster ID: f31b4e3c124846aeac454cc7a613b30c
I20260810 22:36:43.763916 1669 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260810 22:36:43.765524 1669 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fdfd7bce9ad google::LogMessage::Fail() at ??:0
@ 0x7fdfd7bd188a google::LogMessage::SendToLog() at ??:0
@ 0x7fdfd7bce4de google::LogMessage::Flush() at ??:0
@ 0x7fdfd7bcfa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fdfe250dfdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fdfe250ed2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fdfe250b579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fdfe254f640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fdfd9e203d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fdfd9e20c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fdfd9e223f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fdfd9e11512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fdfdb4556db start_thread at ??:0
@ 0x7fdfd66cc61f clone at ??:0
W20260810 22:36:43.865870 1657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:36:43.866202 1657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:36:43.866279 1657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:36:43.870714 1657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:36:43.870821 1657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:36:43.870919 1657 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.65
I20260810 22:36:43.876609 1657 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:33229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab
--principal=kudu/127.16.233.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.65:41675
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.233.65
--webserver_port=37969
--tserver_master_addrs=127.16.233.126:42443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.65
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:36:43.877875 1657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:36:43.879050 1657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:36:43.889995 1703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:36:43.889999 1706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:36:43.890134 1702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:36:43.890687 1657 server_base.cc:1034] running on GCE node
I20260810 22:36:43.891212 1657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:36:43.892119 1657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:36:43.893342 1657 hybrid_clock.cc:648] HybridClock initialized: now 1786401403893311 us; error 41 us; skew 500 ppm
Aug 10 22:36:43 dist-test-slave-77v9 krb5kdc[342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401403, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:36:43.897270 1657 init.cc:377] Logged in from keytab as kudu/127.16.233.65@KRBTEST.COM (short username kudu)
I20260810 22:36:43.898659 1657 webserver.cc:460] Webserver started at http://127.16.233.65:37969/ using document root <none> and password file <none>
I20260810 22:36:43.899488 1657 fs_manager.cc:359] Metadata directory not provided
I20260810 22:36:43.899607 1657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:36:43.904042 1657 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 22:36:43.906677 1713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:36:43.908073 1657 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 22:36:43.908222 1657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-0/wal
uuid: "7285c96084ea4dd5a35d59d5b2b47537"
format_stamp: "Formatted at 2026-08-10 22:36:41 on dist-test-slave-77v9"
server_key: "1b71f0acbd9a94be7fdc3c97ea95ca46"
server_key_iv: "c55487d61b4ce976b69716e17f40e00d"
server_key_version: "encryptionkey@0"
I20260810 22:36:43.908905 1657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:36:43.928026 1657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:36:43.932312 1657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:36:43.932516 1657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:36:43.933320 1657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:36:43.934590 1720 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 22:36:43.940699 1657 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 22:36:43.940819 1657 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260810 22:36:43.940948 1657 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 22:36:43.942975 1657 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 22:36:43.943068 1657 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260810 22:36:43.943415 1720 tablet_bootstrap.cc:492] T 5a1826d942b64ea3ab6cd117740f3d7e P 7285c96084ea4dd5a35d59d5b2b47537: Bootstrap starting.
I20260810 22:36:43.954888 1657 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.65:41675
I20260810 22:36:43.954972 1840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.65:41675 every 8 connection(s)
I20260810 22:36:43.956229 1657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
I20260810 22:36:43.960402 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 1657
I20260810 22:36:43.960984 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.66:41115
--local_ip_for_outbound_sockets=127.16.233.66
--tserver_master_addrs=127.16.233.126:42443
--webserver_port=34011
--webserver_interface=127.16.233.66
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab
--principal=kudu/127.16.233.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:33229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 22:36:43.969317 1720 log.cc:824] T 5a1826d942b64ea3ab6cd117740f3d7e P 7285c96084ea4dd5a35d59d5b2b47537: Log is configured to *not* fsync() on all Append() calls
I20260810 22:36:43.979543 1720 tablet_bootstrap.cc:492] T 5a1826d942b64ea3ab6cd117740f3d7e P 7285c96084ea4dd5a35d59d5b2b47537: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 22:36:43.980219 1720 tablet_bootstrap.cc:492] T 5a1826d942b64ea3ab6cd117740f3d7e P 7285c96084ea4dd5a35d59d5b2b47537: Bootstrap complete.
I20260810 22:36:43.982103 1720 ts_tablet_manager.cc:1397] T 5a1826d942b64ea3ab6cd117740f3d7e P 7285c96084ea4dd5a35d59d5b2b47537: Time spent bootstrapping tablet: real 0.039s user 0.028s sys 0.008s
I20260810 22:36:43.985981 1720 raft_consensus.cc:348] T 5a1826d942b64ea3ab6cd117740f3d7e P 7285c96084ea4dd5a35d59d5b2b47537 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10e4188e49c24a888f122a94d2349393" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 41115 } } peers { permanent_uuid: "89ffcbdb9f8c45c7a748bca58e869e23" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 46119 } } peers { permanent_uuid: "7285c96084ea4dd5a35d59d5b2b47537" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 41675 } }
I20260810 22:36:43.986807 1720 raft_consensus.cc:729] T 5a1826d942b64ea3ab6cd117740f3d7e P 7285c96084ea4dd5a35d59d5b2b47537 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7285c96084ea4dd5a35d59d5b2b47537, State: Initialized, Role: FOLLOWER
I20260810 22:36:43.987524 1720 consensus_queue.cc:260] T 5a1826d942b64ea3ab6cd117740f3d7e P 7285c96084ea4dd5a35d59d5b2b47537 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10e4188e49c24a888f122a94d2349393" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 41115 } } peers { permanent_uuid: "89ffcbdb9f8c45c7a748bca58e869e23" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 46119 } } peers { permanent_uuid: "7285c96084ea4dd5a35d59d5b2b47537" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 41675 } }
I20260810 22:36:43.988897 1720 ts_tablet_manager.cc:1428] T 5a1826d942b64ea3ab6cd117740f3d7e P 7285c96084ea4dd5a35d59d5b2b47537: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260810 22:36:44.081471 1843 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:36:43.958828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:42443 (local address 127.16.233.65:53561)
0810 22:36:43.959499 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:43.959562 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0810 22:36:43.961650 (+ 2088us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:36:44.080337 (+118687us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:42443: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":205,"threads_started":1}
W20260810 22:36:44.082316 1841 heartbeater.cc:644] Failed to heartbeat to 127.16.233.126:42443 (0 consecutive failures): Network error: Failed to ping master at 127.16.233.126:42443: Client connection negotiation failed: client connection to 127.16.233.126:42443: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 22:36:44.083315 1843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:44.082925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:42443 (local address 127.16.233.65:60939)
0810 22:36:44.083138 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:44.083225 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:42443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
I20260810 22:36:44.084115 1843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:44.083842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:42443 (local address 127.16.233.65:57387)
0810 22:36:44.083986 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:44.084036 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:42443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260810 22:36:44.084321 1841 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 22:36:44.111680 1844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:36:44.111994 1844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:36:44.112042 1844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:36:44.116580 1844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:36:44.116667 1844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:36:44.116729 1844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.66
I20260810 22:36:44.123633 1844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:33229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab
--principal=kudu/127.16.233.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.66:41115
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.233.66
--webserver_port=34011
--tserver_master_addrs=127.16.233.126:42443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.66
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:36:44.124943 1844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:36:44.126029 1844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 22:36:44.136931 1844 server_base.cc:1034] running on GCE node
W20260810 22:36:44.136739 1854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:36:44.136739 1852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:36:44.136747 1851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:36:44.137614 1844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:36:44.138371 1844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:36:44.139590 1844 hybrid_clock.cc:648] HybridClock initialized: now 1786401404139578 us; error 39 us; skew 500 ppm
Aug 10 22:36:44 dist-test-slave-77v9 krb5kdc[342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401404, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:36:44.143071 1844 init.cc:377] Logged in from keytab as kudu/127.16.233.66@KRBTEST.COM (short username kudu)
I20260810 22:36:44.144166 1844 webserver.cc:460] Webserver started at http://127.16.233.66:34011/ using document root <none> and password file <none>
I20260810 22:36:44.144891 1844 fs_manager.cc:359] Metadata directory not provided
I20260810 22:36:44.144999 1844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:36:44.149418 1844 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 22:36:44.151837 1863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:36:44.153260 1844 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:36:44.153419 1844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-1/wal
uuid: "10e4188e49c24a888f122a94d2349393"
format_stamp: "Formatted at 2026-08-10 22:36:41 on dist-test-slave-77v9"
server_key: "4492a97e4cb8c53a493e8b2d24fb4488"
server_key_iv: "b3b1b50c3c04d66aafa0cf8eefc69144"
server_key_version: "encryptionkey@0"
I20260810 22:36:44.154053 1844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:36:44.178737 1844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:36:44.182444 1844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:36:44.182684 1844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:36:44.183378 1844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:36:44.184475 1844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:36:44.184576 1844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:36:44.184655 1844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:36:44.184705 1844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:36:44.195116 1844 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.66:41115
I20260810 22:36:44.195138 1987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.66:41115 every 8 connection(s)
I20260810 22:36:44.196260 1844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
I20260810 22:36:44.197357 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 1844
I20260810 22:36:44.197839 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.67:46119
--local_ip_for_outbound_sockets=127.16.233.67
--tserver_master_addrs=127.16.233.126:42443
--webserver_port=42629
--webserver_interface=127.16.233.67
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab
--principal=kudu/127.16.233.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:33229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 22:36:44.201649 1990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:44.198925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:42443 (local address 127.16.233.66:43173)
0810 22:36:44.199526 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:44.199648 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:42443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":161,"threads_started":1}
W20260810 22:36:44.202382 1988 heartbeater.cc:644] Failed to heartbeat to 127.16.233.126:42443 (0 consecutive failures): Network error: Failed to ping master at 127.16.233.126:42443: Client connection negotiation failed: client connection to 127.16.233.126:42443: connect: Connection refused (error 111)
I20260810 22:36:44.203195 1990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:44.202890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:42443 (local address 127.16.233.66:46969)
0810 22:36:44.203056 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:44.203121 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:42443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260810 22:36:44.204035 1990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:44.203782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:42443 (local address 127.16.233.66:52003)
0810 22:36:44.203930 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:44.203977 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:42443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260810 22:36:44.204293 1988 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 22:36:44.320040 1991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:36:44.320376 1991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:36:44.320425 1991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:36:44.324987 1991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:36:44.325079 1991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:36:44.325143 1991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.67
I20260810 22:36:44.330948 1991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:33229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab
--principal=kudu/127.16.233.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.67:46119
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.233.67
--webserver_port=42629
--tserver_master_addrs=127.16.233.126:42443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.67
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:36:44.332295 1991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:36:44.333549 1991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:36:44.344429 1999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:36:44.344409 1996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:36:44.344409 1997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:36:44.344491 1991 server_base.cc:1034] running on GCE node
I20260810 22:36:44.345321 1991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:36:44.346060 1991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:36:44.347263 1991 hybrid_clock.cc:648] HybridClock initialized: now 1786401404347222 us; error 56 us; skew 500 ppm
Aug 10 22:36:44 dist-test-slave-77v9 krb5kdc[342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401404, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:36:44.350577 1991 init.cc:377] Logged in from keytab as kudu/127.16.233.67@KRBTEST.COM (short username kudu)
I20260810 22:36:44.351634 1991 webserver.cc:460] Webserver started at http://127.16.233.67:42629/ using document root <none> and password file <none>
I20260810 22:36:44.352252 1991 fs_manager.cc:359] Metadata directory not provided
I20260810 22:36:44.352352 1991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:36:44.356700 1991 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 22:36:44.359441 2008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:36:44.361080 1991 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:36:44.361245 1991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-2/wal
uuid: "89ffcbdb9f8c45c7a748bca58e869e23"
format_stamp: "Formatted at 2026-08-10 22:36:41 on dist-test-slave-77v9"
server_key: "49282af95a15dc914a459555823d5293"
server_key_iv: "0d44a7994ff90ca1fff763e838475662"
server_key_version: "encryptionkey@0"
I20260810 22:36:44.361924 1991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:36:44.375655 1991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:36:44.379179 1991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:36:44.379424 1991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:36:44.380111 1991 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:36:44.381053 1991 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:36:44.381129 1991 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:36:44.381198 1991 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:36:44.381242 1991 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:36:44.392258 1991 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.67:46119
I20260810 22:36:44.392329 2137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.67:46119 every 8 connection(s)
I20260810 22:36:44.393548 1991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
I20260810 22:36:44.397783 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 1991
I20260810 22:36:44.398252 2140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:44.396125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:42443 (local address 127.16.233.67:42829)
0810 22:36:44.396694 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:44.397320 (+ 626us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:42443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":145,"threads_started":1}
W20260810 22:36:44.398805 2138 heartbeater.cc:644] Failed to heartbeat to 127.16.233.126:42443 (0 consecutive failures): Network error: Failed to ping master at 127.16.233.126:42443: Client connection negotiation failed: client connection to 127.16.233.126:42443: connect: Connection refused (error 111)
I20260810 22:36:44.399572 2140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:44.399305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:42443 (local address 127.16.233.67:53241)
0810 22:36:44.399452 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:44.399505 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:42443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260810 22:36:44.400252 2140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:44.400048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:42443 (local address 127.16.233.67:35671)
0810 22:36:44.400154 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:44.400199 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:42443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260810 22:36:44.400465 2138 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 22:36:45.085523 2142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:45.085049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:42443 (local address 127.16.233.65:36279)
0810 22:36:45.085350 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:45.085415 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:42443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":81,"threads_started":1}
I20260810 22:36:45.205466 2143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:45.205004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:42443 (local address 127.16.233.66:37369)
0810 22:36:45.205289 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:45.205347 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:42443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":86,"threads_started":1}
I20260810 22:36:45.401914 2144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:45.401299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:42443 (local address 127.16.233.67:41481)
0810 22:36:45.401630 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:45.401705 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:42443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":92,"threads_started":1}
I20260810 22:36:45.700474 2145 raft_consensus.cc:482] T 5a1826d942b64ea3ab6cd117740f3d7e P 7285c96084ea4dd5a35d59d5b2b47537 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:36:45.700736 2145 raft_consensus.cc:504] T 5a1826d942b64ea3ab6cd117740f3d7e P 7285c96084ea4dd5a35d59d5b2b47537 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10e4188e49c24a888f122a94d2349393" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 41115 } } peers { permanent_uuid: "89ffcbdb9f8c45c7a748bca58e869e23" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 46119 } } peers { permanent_uuid: "7285c96084ea4dd5a35d59d5b2b47537" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 41675 } }
I20260810 22:36:45.702099 2145 leader_election.cc:288] T 5a1826d942b64ea3ab6cd117740f3d7e P 7285c96084ea4dd5a35d59d5b2b47537 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10e4188e49c24a888f122a94d2349393 (127.16.233.66:41115), 89ffcbdb9f8c45c7a748bca58e869e23 (127.16.233.67:46119)
Aug 10 22:36:45 dist-test-slave-77v9 krb5kdc[342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401403, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.65@KRBTEST.COM for kudu/127.16.233.66@KRBTEST.COM
Aug 10 22:36:45 dist-test-slave-77v9 krb5kdc[342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401403, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.65@KRBTEST.COM for kudu/127.16.233.67@KRBTEST.COM
I20260810 22:36:45.717872 2149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:45.702570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:49571 (local address 127.16.233.66:41115)
0810 22:36:45.703200 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0810 22:36:45.703221 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 22:36:45.703256 (+ 35us) server_negotiation.cc:407] Connection header received
0810 22:36:45.704182 (+ 926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:36:45.704190 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:36:45.704642 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:36:45.705023 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:36:45.706292 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:45.707295 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:45.708598 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:45.709252 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:45.709441 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:45.712403 (+ 2962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:36:45.712408 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:36:45.712421 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:36:45.712454 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:36:45.714696 (+ 2242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:36:45.715349 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:36:45.715356 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:36:45.715360 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:36:45.715505 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:36:45.715918 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:36:45.715928 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:36:45.715931 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:36:45.716407 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:36:45.716605 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0810 22:36:45.717318 (+ 713us) server_negotiation.cc:299] Negotiation successful
0810 22:36:45.717619 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":89,"threads_started":1}
I20260810 22:36:45.717916 2147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:45.702387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.66:41115 (local address 127.16.233.65:49571)
0810 22:36:45.702784 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:45.702801 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 22:36:45.702923 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:36:45.705541 (+ 2618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:36:45.705543 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:36:45.705728 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:36:45.706073 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:45.706092 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:45.707474 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:45.707477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:45.708464 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:45.708475 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:45.709593 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:45.709597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:45.709691 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:45.710257 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:36:45.710282 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:36:45.712149 (+ 1867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:36:45.714864 (+ 2715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:36:45.714871 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:36:45.714893 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:36:45.715193 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:36:45.715645 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:36:45.715649 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:36:45.715650 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:36:45.715802 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:36:45.716588 (+ 786us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:36:45.716595 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:36:45.716962 (+ 367us) client_negotiation.cc:769] Sending connection context
0810 22:36:45.717180 (+ 218us) client_negotiation.cc:240] Negotiation successful
0810 22:36:45.717619 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"mutex_wait_us":250,"spinlock_wait_cycles":896,"thread_start_us":109,"threads_started":1}
I20260810 22:36:45.717924 2150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:45.702602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:58005 (local address 127.16.233.67:46119)
0810 22:36:45.703035 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0810 22:36:45.703065 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0810 22:36:45.703111 (+ 46us) server_negotiation.cc:407] Connection header received
0810 22:36:45.704129 (+ 1018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:36:45.704139 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:36:45.704643 (+ 504us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:36:45.705023 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:36:45.706272 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:45.707295 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:45.708594 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:36:45.709197 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:36:45.709378 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:45.712884 (+ 3506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:36:45.712892 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:36:45.712910 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:36:45.712980 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:36:45.715021 (+ 2041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:36:45.715559 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:36:45.715565 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:36:45.715569 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:36:45.715696 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:36:45.716037 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:36:45.716042 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:36:45.716044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:36:45.716479 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:36:45.716632 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0810 22:36:45.717362 (+ 730us) server_negotiation.cc:299] Negotiation successful
0810 22:36:45.717700 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":130,"threads_started":1}
I20260810 22:36:45.717928 2146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:45.702399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.67:46119 (local address 127.16.233.65:58005)
0810 22:36:45.702854 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:45.702871 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 22:36:45.702973 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:36:45.705097 (+ 2124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:36:45.705102 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:36:45.705513 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:36:45.706075 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:45.706092 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:45.707478 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:45.707481 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:45.708462 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:36:45.708473 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:36:45.709351 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:36:45.709357 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:36:45.709538 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:36:45.710476 (+ 938us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:36:45.710489 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:36:45.712566 (+ 2077us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:36:45.715169 (+ 2603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:36:45.715174 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:36:45.715176 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:36:45.715410 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:36:45.715825 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:36:45.715828 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:36:45.715830 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:36:45.715929 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:36:45.716616 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:36:45.716620 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:36:45.716965 (+ 345us) client_negotiation.cc:769] Sending connection context
0810 22:36:45.717181 (+ 216us) client_negotiation.cc:240] Negotiation successful
0810 22:36:45.717723 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"mutex_wait_us":268,"thread_start_us":69,"threads_started":1}
W20260810 22:36:45.721585 1717 leader_election.cc:341] T 5a1826d942b64ea3ab6cd117740f3d7e P 7285c96084ea4dd5a35d59d5b2b47537 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 89ffcbdb9f8c45c7a748bca58e869e23 (127.16.233.67:46119): Not found: Tablet not found: 5a1826d942b64ea3ab6cd117740f3d7e
W20260810 22:36:45.721897 1714 leader_election.cc:341] T 5a1826d942b64ea3ab6cd117740f3d7e P 7285c96084ea4dd5a35d59d5b2b47537 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 10e4188e49c24a888f122a94d2349393 (127.16.233.66:41115): Not found: Tablet not found: 5a1826d942b64ea3ab6cd117740f3d7e
I20260810 22:36:45.721994 1714 leader_election.cc:302] T 5a1826d942b64ea3ab6cd117740f3d7e P 7285c96084ea4dd5a35d59d5b2b47537 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7285c96084ea4dd5a35d59d5b2b47537; no voters: 10e4188e49c24a888f122a94d2349393, 89ffcbdb9f8c45c7a748bca58e869e23
I20260810 22:36:45.722333 2145 raft_consensus.cc:2726] T 5a1826d942b64ea3ab6cd117740f3d7e P 7285c96084ea4dd5a35d59d5b2b47537 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 22:36:46.086894 2147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:46.086490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:42443 (local address 127.16.233.65:53737)
0810 22:36:46.086706 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:46.086791 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:42443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260810 22:36:46.206980 2151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:46.206413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:42443 (local address 127.16.233.66:47975)
0810 22:36:46.206720 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:46.206790 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:42443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":83,"threads_started":1}
I20260810 22:36:46.403429 2152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:46.402900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:42443 (local address 127.16.233.67:38437)
0810 22:36:46.403229 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:46.403300 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:42443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":89,"threads_started":1}
I20260810 22:36:47.088531 2153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:47.087977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:42443 (local address 127.16.233.65:41591)
0810 22:36:47.088347 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:47.088421 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:42443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":93,"threads_started":1}
I20260810 22:36:47.208448 2155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:47.207991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:42443 (local address 127.16.233.66:34949)
0810 22:36:47.208273 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:47.208344 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:42443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":98,"threads_started":1}
I20260810 22:36:47.405224 2156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:47.404536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:42443 (local address 127.16.233.67:59553)
0810 22:36:47.404970 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:47.405062 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:42443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":93,"threads_started":1}
I20260810 22:36:47.602404 2157 raft_consensus.cc:482] T 5a1826d942b64ea3ab6cd117740f3d7e P 7285c96084ea4dd5a35d59d5b2b47537 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:36:47.602571 2157 raft_consensus.cc:504] T 5a1826d942b64ea3ab6cd117740f3d7e P 7285c96084ea4dd5a35d59d5b2b47537 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10e4188e49c24a888f122a94d2349393" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 41115 } } peers { permanent_uuid: "89ffcbdb9f8c45c7a748bca58e869e23" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 46119 } } peers { permanent_uuid: "7285c96084ea4dd5a35d59d5b2b47537" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 41675 } }
I20260810 22:36:47.603070 2157 leader_election.cc:288] T 5a1826d942b64ea3ab6cd117740f3d7e P 7285c96084ea4dd5a35d59d5b2b47537 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10e4188e49c24a888f122a94d2349393 (127.16.233.66:41115), 89ffcbdb9f8c45c7a748bca58e869e23 (127.16.233.67:46119)
W20260810 22:36:47.603955 1717 leader_election.cc:341] T 5a1826d942b64ea3ab6cd117740f3d7e P 7285c96084ea4dd5a35d59d5b2b47537 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 89ffcbdb9f8c45c7a748bca58e869e23 (127.16.233.67:46119): Not found: Tablet not found: 5a1826d942b64ea3ab6cd117740f3d7e
W20260810 22:36:47.604051 1714 leader_election.cc:341] T 5a1826d942b64ea3ab6cd117740f3d7e P 7285c96084ea4dd5a35d59d5b2b47537 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 10e4188e49c24a888f122a94d2349393 (127.16.233.66:41115): Not found: Tablet not found: 5a1826d942b64ea3ab6cd117740f3d7e
I20260810 22:36:47.604182 1714 leader_election.cc:302] T 5a1826d942b64ea3ab6cd117740f3d7e P 7285c96084ea4dd5a35d59d5b2b47537 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7285c96084ea4dd5a35d59d5b2b47537; no voters: 10e4188e49c24a888f122a94d2349393, 89ffcbdb9f8c45c7a748bca58e869e23
I20260810 22:36:47.604417 2157 raft_consensus.cc:2726] T 5a1826d942b64ea3ab6cd117740f3d7e P 7285c96084ea4dd5a35d59d5b2b47537 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 22:36:48.090023 2158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:48.089517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:42443 (local address 127.16.233.65:46371)
0810 22:36:48.089835 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:48.089918 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:42443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":130,"threads_started":1}
I20260810 22:36:48.210197 2159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:48.209550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:42443 (local address 127.16.233.66:40115)
0810 22:36:48.209962 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:48.210051 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:42443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":97,"threads_started":1}
I20260810 22:36:48.406627 2160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:48.406121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:42443 (local address 127.16.233.67:45941)
0810 22:36:48.406430 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:48.406520 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:42443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":84,"threads_started":1}
I20260810 22:36:49.091698 2161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:49.091164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:42443 (local address 127.16.233.65:46547)
0810 22:36:49.091520 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:49.091592 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:42443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":84,"threads_started":1}
I20260810 22:36:49.211956 2163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:49.211167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:42443 (local address 127.16.233.66:52379)
0810 22:36:49.211513 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:49.211592 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:42443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":108,"threads_started":1}
I20260810 22:36:49.408265 2164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:49.407711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:42443 (local address 127.16.233.67:36219)
0810 22:36:49.408090 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:49.408159 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:42443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":100,"threads_started":1}
I20260810 22:36:50.093473 2165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:50.092941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:42443 (local address 127.16.233.65:50149)
0810 22:36:50.093295 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:50.093362 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:42443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":109,"threads_started":1}
I20260810 22:36:50.213733 2166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:50.213126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:42443 (local address 127.16.233.66:41323)
0810 22:36:50.213535 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:50.213611 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:42443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":111,"threads_started":1}
I20260810 22:36:50.409633 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:50.409121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:42443 (local address 127.16.233.67:55727)
0810 22:36:50.409430 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:50.409502 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:42443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":96,"threads_started":1}
I20260810 22:36:50.495592 2168 raft_consensus.cc:482] T 5a1826d942b64ea3ab6cd117740f3d7e P 7285c96084ea4dd5a35d59d5b2b47537 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:36:50.495817 2168 raft_consensus.cc:504] T 5a1826d942b64ea3ab6cd117740f3d7e P 7285c96084ea4dd5a35d59d5b2b47537 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10e4188e49c24a888f122a94d2349393" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 41115 } } peers { permanent_uuid: "89ffcbdb9f8c45c7a748bca58e869e23" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 46119 } } peers { permanent_uuid: "7285c96084ea4dd5a35d59d5b2b47537" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 41675 } }
I20260810 22:36:50.496376 2168 leader_election.cc:288] T 5a1826d942b64ea3ab6cd117740f3d7e P 7285c96084ea4dd5a35d59d5b2b47537 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10e4188e49c24a888f122a94d2349393 (127.16.233.66:41115), 89ffcbdb9f8c45c7a748bca58e869e23 (127.16.233.67:46119)
W20260810 22:36:50.497396 1714 leader_election.cc:341] T 5a1826d942b64ea3ab6cd117740f3d7e P 7285c96084ea4dd5a35d59d5b2b47537 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 10e4188e49c24a888f122a94d2349393 (127.16.233.66:41115): Not found: Tablet not found: 5a1826d942b64ea3ab6cd117740f3d7e
W20260810 22:36:50.497515 1717 leader_election.cc:341] T 5a1826d942b64ea3ab6cd117740f3d7e P 7285c96084ea4dd5a35d59d5b2b47537 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 89ffcbdb9f8c45c7a748bca58e869e23 (127.16.233.67:46119): Not found: Tablet not found: 5a1826d942b64ea3ab6cd117740f3d7e
I20260810 22:36:50.497645 1717 leader_election.cc:302] T 5a1826d942b64ea3ab6cd117740f3d7e P 7285c96084ea4dd5a35d59d5b2b47537 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7285c96084ea4dd5a35d59d5b2b47537; no voters: 10e4188e49c24a888f122a94d2349393, 89ffcbdb9f8c45c7a748bca58e869e23
I20260810 22:36:50.497926 2168 raft_consensus.cc:2726] T 5a1826d942b64ea3ab6cd117740f3d7e P 7285c96084ea4dd5a35d59d5b2b47537 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 22:36:51.095177 2169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:51.094569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:42443 (local address 127.16.233.65:41711)
0810 22:36:51.094920 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:51.095045 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:42443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":82,"threads_started":1}
I20260810 22:36:51.215210 2170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:51.214653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:42443 (local address 127.16.233.66:33083)
0810 22:36:51.215036 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:51.215108 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:42443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":97,"threads_started":1}
I20260810 22:36:51.411160 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:51.410589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:42443 (local address 127.16.233.67:60573)
0810 22:36:51.410949 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:51.411047 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:42443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":82,"threads_started":1}
I20260810 22:36:52.096640 2173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:52.096129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:42443 (local address 127.16.233.65:51997)
0810 22:36:52.096467 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:52.096536 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:42443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":113,"threads_started":1}
I20260810 22:36:52.216598 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:52.216082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:42443 (local address 127.16.233.66:58549)
0810 22:36:52.216428 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:52.216496 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:42443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":89,"threads_started":1}
I20260810 22:36:52.412672 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:52.412197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:42443 (local address 127.16.233.67:55113)
0810 22:36:52.412497 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:52.412567 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:42443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":81,"threads_started":1}
I20260810 22:36:52.892160 2177 raft_consensus.cc:482] T 5a1826d942b64ea3ab6cd117740f3d7e P 7285c96084ea4dd5a35d59d5b2b47537 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:36:52.892419 2177 raft_consensus.cc:504] T 5a1826d942b64ea3ab6cd117740f3d7e P 7285c96084ea4dd5a35d59d5b2b47537 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10e4188e49c24a888f122a94d2349393" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 41115 } } peers { permanent_uuid: "89ffcbdb9f8c45c7a748bca58e869e23" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 46119 } } peers { permanent_uuid: "7285c96084ea4dd5a35d59d5b2b47537" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 41675 } }
I20260810 22:36:52.893018 2177 leader_election.cc:288] T 5a1826d942b64ea3ab6cd117740f3d7e P 7285c96084ea4dd5a35d59d5b2b47537 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10e4188e49c24a888f122a94d2349393 (127.16.233.66:41115), 89ffcbdb9f8c45c7a748bca58e869e23 (127.16.233.67:46119)
W20260810 22:36:52.894094 1714 leader_election.cc:341] T 5a1826d942b64ea3ab6cd117740f3d7e P 7285c96084ea4dd5a35d59d5b2b47537 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 10e4188e49c24a888f122a94d2349393 (127.16.233.66:41115): Not found: Tablet not found: 5a1826d942b64ea3ab6cd117740f3d7e
W20260810 22:36:52.894237 1717 leader_election.cc:341] T 5a1826d942b64ea3ab6cd117740f3d7e P 7285c96084ea4dd5a35d59d5b2b47537 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 89ffcbdb9f8c45c7a748bca58e869e23 (127.16.233.67:46119): Not found: Tablet not found: 5a1826d942b64ea3ab6cd117740f3d7e
I20260810 22:36:52.894332 1717 leader_election.cc:302] T 5a1826d942b64ea3ab6cd117740f3d7e P 7285c96084ea4dd5a35d59d5b2b47537 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7285c96084ea4dd5a35d59d5b2b47537; no voters: 10e4188e49c24a888f122a94d2349393, 89ffcbdb9f8c45c7a748bca58e869e23
I20260810 22:36:52.894577 2177 raft_consensus.cc:2726] T 5a1826d942b64ea3ab6cd117740f3d7e P 7285c96084ea4dd5a35d59d5b2b47537 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 22:36:53.098282 2178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:53.097748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:42443 (local address 127.16.233.65:50029)
0810 22:36:53.098078 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:53.098140 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:42443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":121,"threads_started":1}
I20260810 22:36:53.218077 2179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:53.217579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:42443 (local address 127.16.233.66:59933)
0810 22:36:53.217903 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:53.217973 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:42443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":103,"threads_started":1}
I20260810 22:36:53.414155 2180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:53.413544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:42443 (local address 127.16.233.67:38891)
0810 22:36:53.413989 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:53.414051 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:42443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":245,"threads_started":1}
I20260810 22:36:54.099895 2181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:54.099330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:42443 (local address 127.16.233.65:47577)
0810 22:36:54.099658 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:54.099731 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:42443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":104,"threads_started":1}
I20260810 22:36:54.219501 2182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:54.219013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:42443 (local address 127.16.233.66:56807)
0810 22:36:54.219306 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:54.219368 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:42443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":106,"threads_started":1}
I20260810 22:36:54.415378 2183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:54.414927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:42443 (local address 127.16.233.67:60895)
0810 22:36:54.415209 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:54.415267 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:42443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":76,"threads_started":1}
I20260810 22:36:55.101984 2184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:55.101299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:42443 (local address 127.16.233.65:44001)
0810 22:36:55.101791 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:55.101859 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:42443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":251,"threads_started":1}
I20260810 22:36:55.220772 2185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:55.220340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:42443 (local address 127.16.233.66:38761)
0810 22:36:55.220609 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:55.220673 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:42443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":85,"threads_started":1}
I20260810 22:36:55.416942 2186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:55.416357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:42443 (local address 127.16.233.67:53653)
0810 22:36:55.416647 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:55.416712 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:42443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":87,"threads_started":1}
I20260810 22:36:56.103523 2187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:56.103003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:42443 (local address 127.16.233.65:43707)
0810 22:36:56.103337 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:56.103408 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:42443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":92,"threads_started":1}
I20260810 22:36:56.222173 2188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:56.221683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:42443 (local address 127.16.233.66:48651)
0810 22:36:56.222007 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:56.222071 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:42443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":122,"threads_started":1}
I20260810 22:36:56.418256 2189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:56.417802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:42443 (local address 127.16.233.67:54965)
0810 22:36:56.418088 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:56.418154 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:42443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":81,"threads_started":1}
I20260810 22:36:56.865792 2190 raft_consensus.cc:482] T 5a1826d942b64ea3ab6cd117740f3d7e P 7285c96084ea4dd5a35d59d5b2b47537 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:36:56.865957 2190 raft_consensus.cc:504] T 5a1826d942b64ea3ab6cd117740f3d7e P 7285c96084ea4dd5a35d59d5b2b47537 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10e4188e49c24a888f122a94d2349393" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 41115 } } peers { permanent_uuid: "89ffcbdb9f8c45c7a748bca58e869e23" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 46119 } } peers { permanent_uuid: "7285c96084ea4dd5a35d59d5b2b47537" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 41675 } }
I20260810 22:36:56.866372 2190 leader_election.cc:288] T 5a1826d942b64ea3ab6cd117740f3d7e P 7285c96084ea4dd5a35d59d5b2b47537 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10e4188e49c24a888f122a94d2349393 (127.16.233.66:41115), 89ffcbdb9f8c45c7a748bca58e869e23 (127.16.233.67:46119)
W20260810 22:36:56.867378 1717 leader_election.cc:341] T 5a1826d942b64ea3ab6cd117740f3d7e P 7285c96084ea4dd5a35d59d5b2b47537 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 89ffcbdb9f8c45c7a748bca58e869e23 (127.16.233.67:46119): Not found: Tablet not found: 5a1826d942b64ea3ab6cd117740f3d7e
W20260810 22:36:56.867496 1714 leader_election.cc:341] T 5a1826d942b64ea3ab6cd117740f3d7e P 7285c96084ea4dd5a35d59d5b2b47537 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 10e4188e49c24a888f122a94d2349393 (127.16.233.66:41115): Not found: Tablet not found: 5a1826d942b64ea3ab6cd117740f3d7e
I20260810 22:36:56.867626 1714 leader_election.cc:302] T 5a1826d942b64ea3ab6cd117740f3d7e P 7285c96084ea4dd5a35d59d5b2b47537 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7285c96084ea4dd5a35d59d5b2b47537; no voters: 10e4188e49c24a888f122a94d2349393, 89ffcbdb9f8c45c7a748bca58e869e23
I20260810 22:36:56.867857 2190 raft_consensus.cc:2726] T 5a1826d942b64ea3ab6cd117740f3d7e P 7285c96084ea4dd5a35d59d5b2b47537 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 22:36:57.105118 2191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:57.104540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:42443 (local address 127.16.233.65:48943)
0810 22:36:57.104934 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:57.105005 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:42443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":95,"threads_started":1}
I20260810 22:36:57.223832 2192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:57.223126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:42443 (local address 127.16.233.66:35891)
0810 22:36:57.223642 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:57.223712 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:42443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":201,"threads_started":1}
I20260810 22:36:57.419589 2193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:57.419148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:42443 (local address 127.16.233.67:52051)
0810 22:36:57.419426 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:57.419487 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:42443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":80,"threads_started":1}
I20260810 22:36:58.106539 2194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:58.106024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:42443 (local address 127.16.233.65:47215)
0810 22:36:58.106364 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:58.106428 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:42443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":108,"threads_started":1}
I20260810 22:36:58.225194 2195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:58.224693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:42443 (local address 127.16.233.66:41013)
0810 22:36:58.225023 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:58.225091 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:42443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":148,"threads_started":1}
I20260810 22:36:58.421013 2198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:58.420484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:42443 (local address 127.16.233.67:36093)
0810 22:36:58.420840 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:58.420909 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:42443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":87,"threads_started":1}
I20260810 22:36:59.108428 2199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:59.107904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:42443 (local address 127.16.233.65:42555)
0810 22:36:59.108221 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:59.108299 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:42443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":87,"threads_started":1}
I20260810 22:36:59.226748 2200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:59.226273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:42443 (local address 127.16.233.66:45303)
0810 22:36:59.226572 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:59.226644 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:42443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":84,"threads_started":1}
I20260810 22:36:59.410094 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 1657
I20260810 22:36:59.419914 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 1844
I20260810 22:36:59.423485 2201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:36:59.422251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:42443 (local address 127.16.233.67:52111)
0810 22:36:59.423326 (+ 1075us) negotiation.cc:107] Waiting for socket to connect
0810 22:36:59.423387 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:42443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":982,"thread_start_us":79,"threads_started":1}
I20260810 22:36:59.429593 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 1991
2026-08-10T22:36:59Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (20757 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:37:00 dist-test-slave-77v9 krb5kdc[2204](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:37:00 dist-test-slave-77v9 krb5kdc[2204](info): set up 2 sockets
Aug 10 22:37:00 dist-test-slave-77v9 krb5kdc[2204](info): commencing operation
krb5kdc: starting...
W20260810 22:37:01.139091 17317 mini_kdc.cc:121] Time spent starting KDC: real 1.107s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Aug 10 22:37:01 dist-test-slave-77v9 krb5kdc[2204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:37:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:37:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.233.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab.
Entry for principal kudu/127.16.233.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab.
I20260810 22:37:01.289801 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:39115
--webserver_interface=127.16.233.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab
--principal=kudu/127.16.233.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:38849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.126:39115
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:37:01.411288 2219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:37:01.411662 2219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:37:01.411742 2219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:37:01.416055 2219 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 22:37:01.416174 2219 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 22:37:01.416203 2219 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 22:37:01.416224 2219 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:37:01.416266 2219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:37:01.416328 2219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:37:01.416364 2219 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:37:01.416402 2219 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:37:01.422106 2219 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:38849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.16.233.126:39115
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab
--principal=kudu/127.16.233.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:39115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2219
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:37:01.423328 2219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:37:01.424412 2219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:37:01.434662 2228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:37:01.434662 2224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:37:01.434711 2219 server_base.cc:1034] running on GCE node
W20260810 22:37:01.434763 2226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:37:01.436082 2219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:37:01.437630 2219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:37:01.438839 2219 hybrid_clock.cc:648] HybridClock initialized: now 1786401421438819 us; error 43 us; skew 500 ppm
Aug 10 22:37:01 dist-test-slave-77v9 krb5kdc[2204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401421, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:37:01.442126 2219 init.cc:377] Logged in from keytab as kudu/127.16.233.126@KRBTEST.COM (short username kudu)
I20260810 22:37:01.443172 2219 webserver.cc:460] Webserver started at http://127.16.233.126:43295/ using document root <none> and password file <none>
I20260810 22:37:01.443784 2219 fs_manager.cc:359] Metadata directory not provided
I20260810 22:37:01.443853 2219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:37:01.444092 2219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:37:01.446412 2219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/master-0/data/instance:
uuid: "c7109630f3354b35aa075a1f467262d4"
format_stamp: "Formatted at 2026-08-10 22:37:01 on dist-test-slave-77v9"
server_key: "2eff5433112c9d11164a106669587b59"
server_key_iv: "8d8791d9ad2768b560b36d7faa4cbe00"
server_key_version: "encryptionkey@0"
I20260810 22:37:01.447069 2219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/master-0/wal/instance:
uuid: "c7109630f3354b35aa075a1f467262d4"
format_stamp: "Formatted at 2026-08-10 22:37:01 on dist-test-slave-77v9"
server_key: "2eff5433112c9d11164a106669587b59"
server_key_iv: "8d8791d9ad2768b560b36d7faa4cbe00"
server_key_version: "encryptionkey@0"
I20260810 22:37:01.451293 2219 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 22:37:01.454053 2236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:37:01.455427 2219 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:37:01.455621 2219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/master-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/master-0/wal
uuid: "c7109630f3354b35aa075a1f467262d4"
format_stamp: "Formatted at 2026-08-10 22:37:01 on dist-test-slave-77v9"
server_key: "2eff5433112c9d11164a106669587b59"
server_key_iv: "8d8791d9ad2768b560b36d7faa4cbe00"
server_key_version: "encryptionkey@0"
I20260810 22:37:01.455750 2219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:37:01.477695 2219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:37:01.481030 2219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:37:01.481256 2219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:37:01.489352 2219 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.126:39115
I20260810 22:37:01.489714 2294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.126:39115 every 8 connection(s)
I20260810 22:37:01.490450 2219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
I20260810 22:37:01.493732 2295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:37:01.498848 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 2219
I20260810 22:37:01.499043 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/master-0/wal/instance
I20260810 22:37:01.499333 17317 external_mini_cluster.cc:1428] Setting key 04d57e193b06b73b3c603a4c43725173
I20260810 22:37:01.500424 2295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7109630f3354b35aa075a1f467262d4: Bootstrap starting.
I20260810 22:37:01.503649 2295 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7109630f3354b35aa075a1f467262d4: Neither blocks nor log segments found. Creating new log.
I20260810 22:37:01.504639 2295 log.cc:824] T 00000000000000000000000000000000 P c7109630f3354b35aa075a1f467262d4: Log is configured to *not* fsync() on all Append() calls
I20260810 22:37:01.507225 2295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7109630f3354b35aa075a1f467262d4: No bootstrap required, opened a new log
Aug 10 22:37:01 dist-test-slave-77v9 krb5kdc[2204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:37:01.512301 2295 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7109630f3354b35aa075a1f467262d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7109630f3354b35aa075a1f467262d4" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39115 } }
I20260810 22:37:01.512663 2295 raft_consensus.cc:374] T 00000000000000000000000000000000 P c7109630f3354b35aa075a1f467262d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:37:01.512741 2295 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7109630f3354b35aa075a1f467262d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7109630f3354b35aa075a1f467262d4, State: Initialized, Role: FOLLOWER
I20260810 22:37:01.513379 2295 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7109630f3354b35aa075a1f467262d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7109630f3354b35aa075a1f467262d4" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39115 } }
I20260810 22:37:01.513561 2295 raft_consensus.cc:388] T 00000000000000000000000000000000 P c7109630f3354b35aa075a1f467262d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:37:01.513613 2295 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7109630f3354b35aa075a1f467262d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:37:01.513731 2295 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7109630f3354b35aa075a1f467262d4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:37:01.515211 2295 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7109630f3354b35aa075a1f467262d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7109630f3354b35aa075a1f467262d4" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39115 } }
I20260810 22:37:01.515650 2295 leader_election.cc:302] T 00000000000000000000000000000000 P c7109630f3354b35aa075a1f467262d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c7109630f3354b35aa075a1f467262d4; no voters:
I20260810 22:37:01.516134 2295 leader_election.cc:288] T 00000000000000000000000000000000 P c7109630f3354b35aa075a1f467262d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:37:01.516227 2300 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7109630f3354b35aa075a1f467262d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:37:01.516469 2300 raft_consensus.cc:686] T 00000000000000000000000000000000 P c7109630f3354b35aa075a1f467262d4 [term 1 LEADER]: Becoming Leader. State: Replica: c7109630f3354b35aa075a1f467262d4, State: Running, Role: LEADER
I20260810 22:37:01.516929 2300 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7109630f3354b35aa075a1f467262d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7109630f3354b35aa075a1f467262d4" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39115 } }
I20260810 22:37:01.517099 2298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:01.500965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40414 (local address 127.16.233.126:39115)
0810 22:37:01.501458 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0810 22:37:01.501480 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 22:37:01.501528 (+ 48us) server_negotiation.cc:407] Connection header received
0810 22:37:01.502542 (+ 1014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:37:01.502569 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:37:01.503063 (+ 494us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:37:01.503484 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:37:01.504469 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:37:01.506057 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:37:01.507014 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:37:01.507822 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:37:01.508019 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:37:01.510828 (+ 2809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:37:01.510836 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:37:01.510858 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:37:01.510906 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:37:01.513284 (+ 2378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:37:01.513963 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:37:01.513973 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:37:01.513983 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:37:01.514161 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:37:01.514454 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:37:01.514458 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:37:01.514460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:37:01.515095 (+ 635us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:37:01.515317 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0810 22:37:01.515648 (+ 331us) server_negotiation.cc:299] Negotiation successful
0810 22:37:01.515989 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":128,"threads_started":1}
I20260810 22:37:01.517669 2295 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7109630f3354b35aa075a1f467262d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:37:01.518815 2301 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7109630f3354b35aa075a1f467262d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c7109630f3354b35aa075a1f467262d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7109630f3354b35aa075a1f467262d4" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39115 } } }
I20260810 22:37:01.518980 2301 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7109630f3354b35aa075a1f467262d4 [sys.catalog]: This master's current role is: LEADER
I20260810 22:37:01.519316 2302 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7109630f3354b35aa075a1f467262d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7109630f3354b35aa075a1f467262d4. Latest consensus state: current_term: 1 leader_uuid: "c7109630f3354b35aa075a1f467262d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7109630f3354b35aa075a1f467262d4" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39115 } } }
I20260810 22:37:01.519410 2302 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7109630f3354b35aa075a1f467262d4 [sys.catalog]: This master's current role is: LEADER
I20260810 22:37:01.520489 2311 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:37:01.523694 2311 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:37:01.530354 2311 catalog_manager.cc:1347] Generated new cluster ID: b3ab420c4a7442a2bc0fcda8f9d00e2b
I20260810 22:37:01.530443 2311 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:37:01.541673 2311 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:37:01.543104 2311 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:37:01.559317 2311 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c7109630f3354b35aa075a1f467262d4: Generated new TSK 0
I20260810 22:37:01.560135 2311 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.233.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab.
Entry for principal kudu/127.16.233.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab.
I20260810 22:37:01.598347 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.65:0
--local_ip_for_outbound_sockets=127.16.233.65
--webserver_interface=127.16.233.65
--webserver_port=0
--tserver_master_addrs=127.16.233.126:39115
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab
--principal=kudu/127.16.233.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:38849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:37:01.725638 2324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:37:01.725984 2324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:37:01.726032 2324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:37:01.730268 2324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:37:01.730350 2324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:37:01.730412 2324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.65
I20260810 22:37:01.736326 2324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:38849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab
--principal=kudu/127.16.233.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.233.65
--webserver_port=0
--tserver_master_addrs=127.16.233.126:39115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.65
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:37:01.737510 2324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:37:01.738526 2324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:37:01.748876 2333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:37:01.748888 2330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:37:01.749017 2331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:37:01.749236 2324 server_base.cc:1034] running on GCE node
I20260810 22:37:01.749729 2324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:37:01.750509 2324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:37:01.751710 2324 hybrid_clock.cc:648] HybridClock initialized: now 1786401421751702 us; error 31 us; skew 500 ppm
Aug 10 22:37:01 dist-test-slave-77v9 krb5kdc[2204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401421, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:37:01.754870 2324 init.cc:377] Logged in from keytab as kudu/127.16.233.65@KRBTEST.COM (short username kudu)
I20260810 22:37:01.755822 2324 webserver.cc:460] Webserver started at http://127.16.233.65:38839/ using document root <none> and password file <none>
I20260810 22:37:01.756421 2324 fs_manager.cc:359] Metadata directory not provided
I20260810 22:37:01.756541 2324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:37:01.756790 2324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:37:01.759020 2324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-0/data/instance:
uuid: "f7df0146ce9c40c9a2d24d9834a2125f"
format_stamp: "Formatted at 2026-08-10 22:37:01 on dist-test-slave-77v9"
server_key: "b284566687ddcb1a6eca3bfc282fa8db"
server_key_iv: "f6cb8a8b6cbd9908e77830f03c31c8c3"
server_key_version: "encryptionkey@0"
I20260810 22:37:01.759609 2324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-0/wal/instance:
uuid: "f7df0146ce9c40c9a2d24d9834a2125f"
format_stamp: "Formatted at 2026-08-10 22:37:01 on dist-test-slave-77v9"
server_key: "b284566687ddcb1a6eca3bfc282fa8db"
server_key_iv: "f6cb8a8b6cbd9908e77830f03c31c8c3"
server_key_version: "encryptionkey@0"
I20260810 22:37:01.763707 2324 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260810 22:37:01.766427 2342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:37:01.767764 2324 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 22:37:01.767915 2324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-0/wal
uuid: "f7df0146ce9c40c9a2d24d9834a2125f"
format_stamp: "Formatted at 2026-08-10 22:37:01 on dist-test-slave-77v9"
server_key: "b284566687ddcb1a6eca3bfc282fa8db"
server_key_iv: "f6cb8a8b6cbd9908e77830f03c31c8c3"
server_key_version: "encryptionkey@0"
I20260810 22:37:01.768054 2324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:37:01.793577 2324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:37:01.797226 2324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:37:01.797490 2324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:37:01.798199 2324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:37:01.799152 2324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:37:01.799229 2324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:37:01.799300 2324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:37:01.799355 2324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:37:01.809592 2324 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.65:36229
I20260810 22:37:01.809648 2468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.65:36229 every 8 connection(s)
I20260810 22:37:01.810604 2324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
I20260810 22:37:01.817533 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 2324
I20260810 22:37:01.817706 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-0/wal/instance
I20260810 22:37:01.818028 17317 external_mini_cluster.cc:1428] Setting key 98ae7c4cadf7e13044e011d6020582f1
Aug 10 22:37:01 dist-test-slave-77v9 krb5kdc[2204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401421, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.65@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:37:01.827230 2298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:01.812618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:36845 (local address 127.16.233.126:39115)
0810 22:37:01.812777 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 22:37:01.812781 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:37:01.813730 (+ 949us) server_negotiation.cc:407] Connection header received
0810 22:37:01.815136 (+ 1406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:37:01.815139 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:37:01.815204 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:37:01.815326 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:37:01.816626 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:37:01.817330 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:37:01.818945 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:37:01.819293 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:37:01.819418 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:37:01.822664 (+ 3246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:37:01.822668 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:37:01.822671 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:37:01.822708 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:37:01.824333 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:37:01.825170 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:37:01.825176 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:37:01.825182 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:37:01.825275 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:37:01.825776 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:37:01.825779 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:37:01.825780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:37:01.825990 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:37:01.826071 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 22:37:01.826942 (+ 871us) server_negotiation.cc:299] Negotiation successful
0810 22:37:01.827096 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 22:37:01.828622 2471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:01.813045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39115 (local address 127.16.233.65:36845)
0810 22:37:01.813557 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0810 22:37:01.813603 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 22:37:01.814895 (+ 1292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:37:01.815517 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:37:01.815524 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:37:01.815939 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:37:01.816455 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:37:01.816469 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:37:01.817479 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:37:01.817482 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:37:01.818795 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:37:01.818805 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:37:01.819480 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:37:01.819485 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:37:01.819737 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:37:01.820457 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:37:01.820484 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:37:01.822464 (+ 1980us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:37:01.824485 (+ 2021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:37:01.824494 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:37:01.824516 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:37:01.825039 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:37:01.825384 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:37:01.825390 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:37:01.825393 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:37:01.825654 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:37:01.826113 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:37:01.826123 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:37:01.826619 (+ 496us) client_negotiation.cc:769] Sending connection context
0810 22:37:01.826893 (+ 274us) client_negotiation.cc:240] Negotiation successful
0810 22:37:01.827338 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":122,"threads_started":1}
WARNING: no policy specified for kudu/127.16.233.66@KRBTEST.COM; defaulting to no policy
I20260810 22:37:01.830160 2469 heartbeater.cc:347] Connected to a master server at 127.16.233.126:39115
I20260810 22:37:01.830479 2469 heartbeater.cc:464] Registering TS with master...
I20260810 22:37:01.831125 2469 heartbeater.cc:511] Master 127.16.233.126:39115 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.66@KRBTEST.COM" created.
I20260810 22:37:01.833056 2251 ts_manager.cc:194] Registered new tserver with Master: f7df0146ce9c40c9a2d24d9834a2125f (127.16.233.65:36229)
I20260810 22:37:01.834442 2251 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.65@KRBTEST.COM'} at 127.16.233.65:36845
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab.
Entry for principal kudu/127.16.233.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab.
I20260810 22:37:01.852660 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.66:0
--local_ip_for_outbound_sockets=127.16.233.66
--webserver_interface=127.16.233.66
--webserver_port=0
--tserver_master_addrs=127.16.233.126:39115
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab
--principal=kudu/127.16.233.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:38849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:37:01.974270 2475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:37:01.974606 2475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:37:01.974689 2475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:37:01.979012 2475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:37:01.979126 2475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:37:01.979228 2475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.66
I20260810 22:37:01.984993 2475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:38849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab
--principal=kudu/127.16.233.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.233.66
--webserver_port=0
--tserver_master_addrs=127.16.233.126:39115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.66
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:37:01.986325 2475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:37:01.987620 2475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:37:01.998188 2484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:37:01.998234 2480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:37:01.998231 2481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:37:01.998281 2475 server_base.cc:1034] running on GCE node
I20260810 22:37:01.998916 2475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:37:01.999655 2475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:37:02.000867 2475 hybrid_clock.cc:648] HybridClock initialized: now 1786401422000834 us; error 37 us; skew 500 ppm
Aug 10 22:37:02 dist-test-slave-77v9 krb5kdc[2204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401422, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:37:02.003906 2475 init.cc:377] Logged in from keytab as kudu/127.16.233.66@KRBTEST.COM (short username kudu)
I20260810 22:37:02.004897 2475 webserver.cc:460] Webserver started at http://127.16.233.66:41565/ using document root <none> and password file <none>
I20260810 22:37:02.005535 2475 fs_manager.cc:359] Metadata directory not provided
I20260810 22:37:02.005633 2475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:37:02.005864 2475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:37:02.008055 2475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-1/data/instance:
uuid: "32bc0a72e0f7471186c4b04c2e8a0422"
format_stamp: "Formatted at 2026-08-10 22:37:02 on dist-test-slave-77v9"
server_key: "90aac31c1a5f558d2a48a0697537b93a"
server_key_iv: "f08935cd48ca5afccf11d200362e9c96"
server_key_version: "encryptionkey@0"
I20260810 22:37:02.008692 2475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-1/wal/instance:
uuid: "32bc0a72e0f7471186c4b04c2e8a0422"
format_stamp: "Formatted at 2026-08-10 22:37:02 on dist-test-slave-77v9"
server_key: "90aac31c1a5f558d2a48a0697537b93a"
server_key_iv: "f08935cd48ca5afccf11d200362e9c96"
server_key_version: "encryptionkey@0"
I20260810 22:37:02.013053 2475 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 22:37:02.015758 2491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:37:02.017350 2475 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:37:02.017498 2475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-1/wal
uuid: "32bc0a72e0f7471186c4b04c2e8a0422"
format_stamp: "Formatted at 2026-08-10 22:37:02 on dist-test-slave-77v9"
server_key: "90aac31c1a5f558d2a48a0697537b93a"
server_key_iv: "f08935cd48ca5afccf11d200362e9c96"
server_key_version: "encryptionkey@0"
I20260810 22:37:02.017661 2475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:37:02.046206 2475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:37:02.049853 2475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:37:02.050097 2475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:37:02.050745 2475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:37:02.051698 2475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:37:02.051772 2475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:37:02.051843 2475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:37:02.051892 2475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:37:02.062065 2475 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.66:34433
I20260810 22:37:02.062111 2615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.66:34433 every 8 connection(s)
I20260810 22:37:02.063086 2475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
I20260810 22:37:02.071651 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 2475
I20260810 22:37:02.071789 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-1/wal/instance
I20260810 22:37:02.072113 17317 external_mini_cluster.cc:1428] Setting key ba80e93630757fa700628a435f1d9310
Aug 10 22:37:02 dist-test-slave-77v9 krb5kdc[2204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401422, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.66@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:37:02.080044 2298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:02.065276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:54165 (local address 127.16.233.126:39115)
0810 22:37:02.065430 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 22:37:02.065436 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:37:02.066404 (+ 968us) server_negotiation.cc:407] Connection header received
0810 22:37:02.067842 (+ 1438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:37:02.067845 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:37:02.067905 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:37:02.067999 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:37:02.069515 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:37:02.070126 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:37:02.071518 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:37:02.071887 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:37:02.072008 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:37:02.075297 (+ 3289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:37:02.075304 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:37:02.075311 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:37:02.075350 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:37:02.077217 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:37:02.077932 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:37:02.077936 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:37:02.077937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:37:02.078040 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:37:02.078499 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:37:02.078503 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:37:02.078504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:37:02.078779 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:37:02.078871 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 22:37:02.079708 (+ 837us) server_negotiation.cc:299] Negotiation successful
0810 22:37:02.079894 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 22:37:02.081447 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:02.065742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39115 (local address 127.16.233.66:54165)
0810 22:37:02.066225 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0810 22:37:02.066270 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 22:37:02.067554 (+ 1284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:37:02.068230 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:37:02.068241 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:37:02.068655 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:37:02.069342 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:37:02.069357 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:37:02.070271 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:37:02.070275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:37:02.071324 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:37:02.071335 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:37:02.072110 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:37:02.072114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:37:02.072349 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:37:02.073215 (+ 866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:37:02.073235 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:37:02.075105 (+ 1870us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:37:02.077348 (+ 2243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:37:02.077357 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:37:02.077379 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:37:02.077795 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:37:02.078155 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:37:02.078161 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:37:02.078166 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:37:02.078375 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:37:02.078912 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:37:02.078921 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:37:02.079338 (+ 417us) client_negotiation.cc:769] Sending connection context
0810 22:37:02.079617 (+ 279us) client_negotiation.cc:240] Negotiation successful
0810 22:37:02.080051 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":122,"threads_started":1}
I20260810 22:37:02.082923 2617 heartbeater.cc:347] Connected to a master server at 127.16.233.126:39115
I20260810 22:37:02.083227 2617 heartbeater.cc:464] Registering TS with master...
I20260810 22:37:02.083801 2617 heartbeater.cc:511] Master 127.16.233.126:39115 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.16.233.67@KRBTEST.COM; defaulting to no policy
I20260810 22:37:02.085145 2251 ts_manager.cc:194] Registered new tserver with Master: 32bc0a72e0f7471186c4b04c2e8a0422 (127.16.233.66:34433)
I20260810 22:37:02.085832 2251 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.66@KRBTEST.COM'} at 127.16.233.66:54165
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab.
Entry for principal kudu/127.16.233.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab.
I20260810 22:37:02.109113 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.67:0
--local_ip_for_outbound_sockets=127.16.233.67
--webserver_interface=127.16.233.67
--webserver_port=0
--tserver_master_addrs=127.16.233.126:39115
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab
--principal=kudu/127.16.233.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:38849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:37:02.239080 2623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:37:02.239395 2623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:37:02.239473 2623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:37:02.243659 2623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:37:02.243755 2623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:37:02.243846 2623 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.67
I20260810 22:37:02.249370 2623 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:38849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab
--principal=kudu/127.16.233.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.233.67
--webserver_port=0
--tserver_master_addrs=127.16.233.126:39115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.67
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:37:02.250527 2623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:37:02.251633 2623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:37:02.261665 2631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:37:02.261837 2633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:37:02.261739 2630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:37:02.262465 2623 server_base.cc:1034] running on GCE node
I20260810 22:37:02.262950 2623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:37:02.263759 2623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:37:02.264937 2623 hybrid_clock.cc:648] HybridClock initialized: now 1786401422264900 us; error 49 us; skew 500 ppm
Aug 10 22:37:02 dist-test-slave-77v9 krb5kdc[2204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401422, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:37:02.268244 2623 init.cc:377] Logged in from keytab as kudu/127.16.233.67@KRBTEST.COM (short username kudu)
I20260810 22:37:02.269332 2623 webserver.cc:460] Webserver started at http://127.16.233.67:35907/ using document root <none> and password file <none>
I20260810 22:37:02.269915 2623 fs_manager.cc:359] Metadata directory not provided
I20260810 22:37:02.269980 2623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:37:02.270171 2623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:37:02.272643 2623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-2/data/instance:
uuid: "f1e19b4ab0404fbeb1208e06ed4bf0ee"
format_stamp: "Formatted at 2026-08-10 22:37:02 on dist-test-slave-77v9"
server_key: "b8e0bfe2c8721b4377d9f951121fd9d6"
server_key_iv: "62ce57aae310ed966623d263463c8469"
server_key_version: "encryptionkey@0"
I20260810 22:37:02.273339 2623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-2/wal/instance:
uuid: "f1e19b4ab0404fbeb1208e06ed4bf0ee"
format_stamp: "Formatted at 2026-08-10 22:37:02 on dist-test-slave-77v9"
server_key: "b8e0bfe2c8721b4377d9f951121fd9d6"
server_key_iv: "62ce57aae310ed966623d263463c8469"
server_key_version: "encryptionkey@0"
I20260810 22:37:02.277495 2623 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 22:37:02.280037 2640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:37:02.281493 2623 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:37:02.281618 2623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-2/wal
uuid: "f1e19b4ab0404fbeb1208e06ed4bf0ee"
format_stamp: "Formatted at 2026-08-10 22:37:02 on dist-test-slave-77v9"
server_key: "b8e0bfe2c8721b4377d9f951121fd9d6"
server_key_iv: "62ce57aae310ed966623d263463c8469"
server_key_version: "encryptionkey@0"
I20260810 22:37:02.281726 2623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:37:02.309283 2623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:37:02.312923 2623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:37:02.313103 2623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:37:02.313678 2623 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:37:02.314502 2623 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:37:02.314551 2623 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:37:02.314592 2623 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:37:02.314608 2623 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:37:02.324446 2623 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.67:38143
I20260810 22:37:02.324529 2769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.67:38143 every 8 connection(s)
I20260810 22:37:02.325485 2623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
I20260810 22:37:02.327703 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 2623
I20260810 22:37:02.327850 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-2/wal/instance
I20260810 22:37:02.328184 17317 external_mini_cluster.cc:1428] Setting key 92ca95c8e25831695df3d37b3835f3fc
Aug 10 22:37:02 dist-test-slave-77v9 krb5kdc[2204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401422, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.67@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:37:02.339828 2298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:02.327380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:60467 (local address 127.16.233.126:39115)
0810 22:37:02.327563 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 22:37:02.327569 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:37:02.328566 (+ 997us) server_negotiation.cc:407] Connection header received
0810 22:37:02.330527 (+ 1961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:37:02.330532 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:37:02.330591 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:37:02.330674 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:37:02.331874 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:37:02.332413 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:37:02.333705 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:37:02.333915 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:37:02.334002 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:37:02.336609 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:37:02.336612 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:37:02.336614 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:37:02.336638 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:37:02.337974 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:37:02.338472 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:37:02.338475 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:37:02.338476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:37:02.338553 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:37:02.338850 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:37:02.338852 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:37:02.338853 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:37:02.339033 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:37:02.339111 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 22:37:02.339590 (+ 479us) server_negotiation.cc:299] Negotiation successful
0810 22:37:02.339698 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 22:37:02.340818 2772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:02.327797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39115 (local address 127.16.233.67:60467)
0810 22:37:02.328398 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0810 22:37:02.328460 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 22:37:02.330260 (+ 1800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:37:02.330858 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:37:02.330865 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:37:02.331248 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:37:02.331733 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:37:02.331747 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:37:02.332536 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:37:02.332538 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:37:02.333594 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:37:02.333603 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:37:02.334051 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:37:02.334053 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:37:02.334198 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:37:02.334854 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:37:02.334881 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:37:02.336447 (+ 1566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:37:02.338087 (+ 1640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:37:02.338091 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:37:02.338104 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:37:02.338370 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:37:02.338634 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:37:02.338636 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:37:02.338637 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:37:02.338772 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:37:02.339121 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:37:02.339125 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:37:02.339364 (+ 239us) client_negotiation.cc:769] Sending connection context
0810 22:37:02.339526 (+ 162us) client_negotiation.cc:240] Negotiation successful
0810 22:37:02.339785 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":133,"threads_started":1}
I20260810 22:37:02.341998 2770 heartbeater.cc:347] Connected to a master server at 127.16.233.126:39115
I20260810 22:37:02.342293 2770 heartbeater.cc:464] Registering TS with master...
I20260810 22:37:02.342875 2770 heartbeater.cc:511] Master 127.16.233.126:39115 requested a full tablet report, sending...
I20260810 22:37:02.343974 2255 ts_manager.cc:194] Registered new tserver with Master: f1e19b4ab0404fbeb1208e06ed4bf0ee (127.16.233.67:38143)
I20260810 22:37:02.344672 2255 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.67@KRBTEST.COM'} at 127.16.233.67:60467
I20260810 22:37:02.354775 17317 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 22:37:02.365922 2298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:02.357479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40430 (local address 127.16.233.126:39115)
0810 22:37:02.357626 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 22:37:02.357630 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:37:02.357904 (+ 274us) server_negotiation.cc:407] Connection header received
0810 22:37:02.358034 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:37:02.358037 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:37:02.358080 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:37:02.358156 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:37:02.358942 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:37:02.359475 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:37:02.360569 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:37:02.360779 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:37:02.360943 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:37:02.362691 (+ 1748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:37:02.362693 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:37:02.362696 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:37:02.362720 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:37:02.364216 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:37:02.364696 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:37:02.364699 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:37:02.364700 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:37:02.364775 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:37:02.365114 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:37:02.365116 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:37:02.365117 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:37:02.365340 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:37:02.365456 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 22:37:02.365696 (+ 240us) server_negotiation.cc:299] Negotiation successful
0810 22:37:02.365796 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 22:37:02.369467 2255 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40430:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 22:37:02.372678 2255 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:37:02.388492 2788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:02.381623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.66:34433 (local address 127.0.0.1:52216)
0810 22:37:02.382795 (+ 1172us) negotiation.cc:107] Waiting for socket to connect
0810 22:37:02.382812 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 22:37:02.383247 (+ 435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:37:02.383907 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:37:02.383912 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:37:02.383943 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:37:02.384040 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:37:02.384045 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:37:02.385549 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:37:02.385552 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:37:02.386882 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:37:02.386896 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:37:02.388105 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:37:02.388111 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:37:02.388203 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:37:02.388223 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 22:37:02.388287 (+ 64us) client_negotiation.cc:240] Negotiation successful
0810 22:37:02.388338 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1112,"mutex_wait_us":8,"thread_start_us":144,"threads_started":1}
I20260810 22:37:02.388536 2791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:02.382509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37410 (local address 127.16.233.67:38143)
0810 22:37:02.382972 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0810 22:37:02.382984 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 22:37:02.383137 (+ 153us) server_negotiation.cc:407] Connection header received
0810 22:37:02.383459 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:37:02.383464 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:37:02.383598 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:37:02.383746 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:37:02.383979 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:37:02.384766 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:37:02.386774 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:37:02.387637 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:37:02.387760 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:37:02.387933 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0810 22:37:02.388297 (+ 364us) server_negotiation.cc:299] Negotiation successful
0810 22:37:02.388394 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":150,"threads_started":1}
I20260810 22:37:02.388535 2787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:02.381608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39038 (local address 127.16.233.65:36229)
0810 22:37:02.382197 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0810 22:37:02.382205 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 22:37:02.382232 (+ 27us) server_negotiation.cc:407] Connection header received
0810 22:37:02.382319 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:37:02.382328 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:37:02.382748 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:37:02.383418 (+ 670us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:37:02.383560 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:37:02.384907 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:37:02.387043 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:37:02.387672 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:37:02.388023 (+ 351us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:37:02.388188 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0810 22:37:02.388249 (+ 61us) server_negotiation.cc:299] Negotiation successful
0810 22:37:02.388363 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":242,"threads_started":1}
I20260810 22:37:02.388955 2786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:02.382089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.67:38143 (local address 127.0.0.1:37410)
0810 22:37:02.383038 (+ 949us) negotiation.cc:107] Waiting for socket to connect
0810 22:37:02.383049 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 22:37:02.383329 (+ 280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:37:02.383726 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:37:02.383729 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:37:02.383747 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:37:02.383847 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:37:02.383856 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:37:02.384955 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:37:02.384957 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:37:02.386657 (+ 1700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:37:02.386667 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:37:02.387790 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:37:02.387795 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:37:02.387929 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:37:02.388033 (+ 104us) client_negotiation.cc:769] Sending connection context
0810 22:37:02.388157 (+ 124us) client_negotiation.cc:240] Negotiation successful
0810 22:37:02.388790 (+ 633us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":879,"mutex_wait_us":550,"thread_start_us":88,"threads_started":1}
I20260810 22:37:02.388492 2784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:02.381412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.65:36229 (local address 127.0.0.1:39038)
0810 22:37:02.381921 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0810 22:37:02.381985 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0810 22:37:02.382107 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:37:02.382952 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:37:02.382959 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:37:02.382986 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:37:02.383203 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:37:02.383211 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:37:02.385025 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:37:02.385027 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:37:02.386861 (+ 1834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:37:02.386874 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:37:02.387857 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:37:02.387860 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:37:02.388047 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:37:02.388080 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 22:37:02.388145 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 22:37:02.388308 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"mutex_wait_us":394,"thread_start_us":66,"threads_started":1}
I20260810 22:37:02.389235 2790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:02.382223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52216 (local address 127.16.233.66:34433)
0810 22:37:02.382653 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0810 22:37:02.382667 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 22:37:02.382897 (+ 230us) server_negotiation.cc:407] Connection header received
0810 22:37:02.383436 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:37:02.383442 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:37:02.383643 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:37:02.383805 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:37:02.384175 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:37:02.385336 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:37:02.387014 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:37:02.387785 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:37:02.387881 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:37:02.388040 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0810 22:37:02.388958 (+ 918us) server_negotiation.cc:299] Negotiation successful
0810 22:37:02.389064 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":163,"threads_started":1}
I20260810 22:37:02.392011 2394 tablet_service.cc:1467] Processing CreateTablet for tablet 84be25a3db6849fd99efa3182fb27460 (DEFAULT_TABLE table=test-table [id=c9f38f3c4541496b8a4e2975e0e7172b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:37:02.392221 2696 tablet_service.cc:1467] Processing CreateTablet for tablet 84be25a3db6849fd99efa3182fb27460 (DEFAULT_TABLE table=test-table [id=c9f38f3c4541496b8a4e2975e0e7172b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:37:02.392225 2541 tablet_service.cc:1467] Processing CreateTablet for tablet 84be25a3db6849fd99efa3182fb27460 (DEFAULT_TABLE table=test-table [id=c9f38f3c4541496b8a4e2975e0e7172b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:37:02.393168 2394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84be25a3db6849fd99efa3182fb27460. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:37:02.393368 2541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84be25a3db6849fd99efa3182fb27460. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:37:02.393379 2696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84be25a3db6849fd99efa3182fb27460. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:37:02.400223 2792 tablet_bootstrap.cc:492] T 84be25a3db6849fd99efa3182fb27460 P f7df0146ce9c40c9a2d24d9834a2125f: Bootstrap starting.
I20260810 22:37:02.401743 2794 tablet_bootstrap.cc:492] T 84be25a3db6849fd99efa3182fb27460 P 32bc0a72e0f7471186c4b04c2e8a0422: Bootstrap starting.
I20260810 22:37:02.401743 2793 tablet_bootstrap.cc:492] T 84be25a3db6849fd99efa3182fb27460 P f1e19b4ab0404fbeb1208e06ed4bf0ee: Bootstrap starting.
I20260810 22:37:02.403473 2792 tablet_bootstrap.cc:654] T 84be25a3db6849fd99efa3182fb27460 P f7df0146ce9c40c9a2d24d9834a2125f: Neither blocks nor log segments found. Creating new log.
I20260810 22:37:02.404137 2793 tablet_bootstrap.cc:654] T 84be25a3db6849fd99efa3182fb27460 P f1e19b4ab0404fbeb1208e06ed4bf0ee: Neither blocks nor log segments found. Creating new log.
I20260810 22:37:02.404291 2792 log.cc:824] T 84be25a3db6849fd99efa3182fb27460 P f7df0146ce9c40c9a2d24d9834a2125f: Log is configured to *not* fsync() on all Append() calls
I20260810 22:37:02.404776 2794 tablet_bootstrap.cc:654] T 84be25a3db6849fd99efa3182fb27460 P 32bc0a72e0f7471186c4b04c2e8a0422: Neither blocks nor log segments found. Creating new log.
I20260810 22:37:02.405767 2794 log.cc:824] T 84be25a3db6849fd99efa3182fb27460 P 32bc0a72e0f7471186c4b04c2e8a0422: Log is configured to *not* fsync() on all Append() calls
I20260810 22:37:02.406777 2792 tablet_bootstrap.cc:492] T 84be25a3db6849fd99efa3182fb27460 P f7df0146ce9c40c9a2d24d9834a2125f: No bootstrap required, opened a new log
I20260810 22:37:02.406873 2793 log.cc:824] T 84be25a3db6849fd99efa3182fb27460 P f1e19b4ab0404fbeb1208e06ed4bf0ee: Log is configured to *not* fsync() on all Append() calls
I20260810 22:37:02.407015 2792 ts_tablet_manager.cc:1397] T 84be25a3db6849fd99efa3182fb27460 P f7df0146ce9c40c9a2d24d9834a2125f: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260810 22:37:02.408190 2794 tablet_bootstrap.cc:492] T 84be25a3db6849fd99efa3182fb27460 P 32bc0a72e0f7471186c4b04c2e8a0422: No bootstrap required, opened a new log
I20260810 22:37:02.408387 2794 ts_tablet_manager.cc:1397] T 84be25a3db6849fd99efa3182fb27460 P 32bc0a72e0f7471186c4b04c2e8a0422: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 22:37:02.409343 2793 tablet_bootstrap.cc:492] T 84be25a3db6849fd99efa3182fb27460 P f1e19b4ab0404fbeb1208e06ed4bf0ee: No bootstrap required, opened a new log
I20260810 22:37:02.409551 2793 ts_tablet_manager.cc:1397] T 84be25a3db6849fd99efa3182fb27460 P f1e19b4ab0404fbeb1208e06ed4bf0ee: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260810 22:37:02.411131 2792 raft_consensus.cc:348] T 84be25a3db6849fd99efa3182fb27460 P f7df0146ce9c40c9a2d24d9834a2125f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7df0146ce9c40c9a2d24d9834a2125f" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 36229 } } peers { permanent_uuid: "32bc0a72e0f7471186c4b04c2e8a0422" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 34433 } } peers { permanent_uuid: "f1e19b4ab0404fbeb1208e06ed4bf0ee" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 38143 } }
I20260810 22:37:02.411399 2792 raft_consensus.cc:374] T 84be25a3db6849fd99efa3182fb27460 P f7df0146ce9c40c9a2d24d9834a2125f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:37:02.411455 2792 raft_consensus.cc:729] T 84be25a3db6849fd99efa3182fb27460 P f7df0146ce9c40c9a2d24d9834a2125f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7df0146ce9c40c9a2d24d9834a2125f, State: Initialized, Role: FOLLOWER
I20260810 22:37:02.411917 2792 consensus_queue.cc:260] T 84be25a3db6849fd99efa3182fb27460 P f7df0146ce9c40c9a2d24d9834a2125f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7df0146ce9c40c9a2d24d9834a2125f" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 36229 } } peers { permanent_uuid: "32bc0a72e0f7471186c4b04c2e8a0422" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 34433 } } peers { permanent_uuid: "f1e19b4ab0404fbeb1208e06ed4bf0ee" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 38143 } }
I20260810 22:37:02.412628 2469 heartbeater.cc:503] Master 127.16.233.126:39115 was elected leader, sending a full tablet report...
I20260810 22:37:02.412935 2794 raft_consensus.cc:348] T 84be25a3db6849fd99efa3182fb27460 P 32bc0a72e0f7471186c4b04c2e8a0422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7df0146ce9c40c9a2d24d9834a2125f" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 36229 } } peers { permanent_uuid: "32bc0a72e0f7471186c4b04c2e8a0422" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 34433 } } peers { permanent_uuid: "f1e19b4ab0404fbeb1208e06ed4bf0ee" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 38143 } }
I20260810 22:37:02.413249 2794 raft_consensus.cc:374] T 84be25a3db6849fd99efa3182fb27460 P 32bc0a72e0f7471186c4b04c2e8a0422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:37:02.413318 2794 raft_consensus.cc:729] T 84be25a3db6849fd99efa3182fb27460 P 32bc0a72e0f7471186c4b04c2e8a0422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32bc0a72e0f7471186c4b04c2e8a0422, State: Initialized, Role: FOLLOWER
I20260810 22:37:02.413777 2794 consensus_queue.cc:260] T 84be25a3db6849fd99efa3182fb27460 P 32bc0a72e0f7471186c4b04c2e8a0422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7df0146ce9c40c9a2d24d9834a2125f" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 36229 } } peers { permanent_uuid: "32bc0a72e0f7471186c4b04c2e8a0422" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 34433 } } peers { permanent_uuid: "f1e19b4ab0404fbeb1208e06ed4bf0ee" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 38143 } }
I20260810 22:37:02.413829 2793 raft_consensus.cc:348] T 84be25a3db6849fd99efa3182fb27460 P f1e19b4ab0404fbeb1208e06ed4bf0ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7df0146ce9c40c9a2d24d9834a2125f" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 36229 } } peers { permanent_uuid: "32bc0a72e0f7471186c4b04c2e8a0422" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 34433 } } peers { permanent_uuid: "f1e19b4ab0404fbeb1208e06ed4bf0ee" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 38143 } }
I20260810 22:37:02.414108 2793 raft_consensus.cc:374] T 84be25a3db6849fd99efa3182fb27460 P f1e19b4ab0404fbeb1208e06ed4bf0ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:37:02.414165 2793 raft_consensus.cc:729] T 84be25a3db6849fd99efa3182fb27460 P f1e19b4ab0404fbeb1208e06ed4bf0ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e19b4ab0404fbeb1208e06ed4bf0ee, State: Initialized, Role: FOLLOWER
I20260810 22:37:02.414312 2792 ts_tablet_manager.cc:1428] T 84be25a3db6849fd99efa3182fb27460 P f7df0146ce9c40c9a2d24d9834a2125f: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260810 22:37:02.414640 2793 consensus_queue.cc:260] T 84be25a3db6849fd99efa3182fb27460 P f1e19b4ab0404fbeb1208e06ed4bf0ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7df0146ce9c40c9a2d24d9834a2125f" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 36229 } } peers { permanent_uuid: "32bc0a72e0f7471186c4b04c2e8a0422" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 34433 } } peers { permanent_uuid: "f1e19b4ab0404fbeb1208e06ed4bf0ee" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 38143 } }
I20260810 22:37:02.414853 2794 ts_tablet_manager.cc:1428] T 84be25a3db6849fd99efa3182fb27460 P 32bc0a72e0f7471186c4b04c2e8a0422: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 22:37:02.415089 2617 heartbeater.cc:503] Master 127.16.233.126:39115 was elected leader, sending a full tablet report...
I20260810 22:37:02.415952 2793 ts_tablet_manager.cc:1428] T 84be25a3db6849fd99efa3182fb27460 P f1e19b4ab0404fbeb1208e06ed4bf0ee: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 22:37:02.416381 2770 heartbeater.cc:503] Master 127.16.233.126:39115 was elected leader, sending a full tablet report...
I20260810 22:37:02.429055 2801 raft_consensus.cc:482] T 84be25a3db6849fd99efa3182fb27460 P f1e19b4ab0404fbeb1208e06ed4bf0ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:37:02.429337 2801 raft_consensus.cc:504] T 84be25a3db6849fd99efa3182fb27460 P f1e19b4ab0404fbeb1208e06ed4bf0ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7df0146ce9c40c9a2d24d9834a2125f" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 36229 } } peers { permanent_uuid: "32bc0a72e0f7471186c4b04c2e8a0422" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 34433 } } peers { permanent_uuid: "f1e19b4ab0404fbeb1208e06ed4bf0ee" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 38143 } }
I20260810 22:37:02.430665 2801 leader_election.cc:288] T 84be25a3db6849fd99efa3182fb27460 P f1e19b4ab0404fbeb1208e06ed4bf0ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f7df0146ce9c40c9a2d24d9834a2125f (127.16.233.65:36229), 32bc0a72e0f7471186c4b04c2e8a0422 (127.16.233.66:34433)
I20260810 22:37:02.436511 2790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:02.431365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:38391 (local address 127.16.233.66:34433)
0810 22:37:02.431655 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0810 22:37:02.431661 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:37:02.431680 (+ 19us) server_negotiation.cc:407] Connection header received
0810 22:37:02.431751 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:37:02.431756 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:37:02.431827 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:37:02.431921 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:37:02.432278 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:37:02.433238 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:37:02.435368 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:37:02.435892 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:37:02.436036 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:37:02.436093 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 22:37:02.436300 (+ 207us) server_negotiation.cc:299] Negotiation successful
0810 22:37:02.436374 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188}
I20260810 22:37:02.436511 2772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:02.431017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.66:34433 (local address 127.16.233.67:38391)
0810 22:37:02.431222 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0810 22:37:02.431241 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 22:37:02.431372 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:37:02.431955 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:37:02.431958 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:37:02.431974 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:37:02.432138 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:37:02.432145 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:37:02.433441 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:37:02.433445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:37:02.435206 (+ 1761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:37:02.435219 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:37:02.436029 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:37:02.436033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:37:02.436158 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:37:02.436177 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 22:37:02.436283 (+ 106us) client_negotiation.cc:240] Negotiation successful
0810 22:37:02.436373 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"mutex_wait_us":146}
I20260810 22:37:02.436703 2787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:02.431176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:34897 (local address 127.16.233.65:36229)
0810 22:37:02.431594 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0810 22:37:02.431600 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:37:02.431800 (+ 200us) server_negotiation.cc:407] Connection header received
0810 22:37:02.431942 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:37:02.431950 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:37:02.432013 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:37:02.432099 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:37:02.432432 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:37:02.433292 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:37:02.435410 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:37:02.436021 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:37:02.436179 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:37:02.436239 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 22:37:02.436474 (+ 235us) server_negotiation.cc:299] Negotiation successful
0810 22:37:02.436569 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284}
I20260810 22:37:02.436728 2802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:02.431017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.65:36229 (local address 127.16.233.67:34897)
0810 22:37:02.431672 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0810 22:37:02.431688 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 22:37:02.431798 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:37:02.432132 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:37:02.432135 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:37:02.432152 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:37:02.432288 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:37:02.432302 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:37:02.433441 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:37:02.433445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:37:02.435208 (+ 1763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:37:02.435226 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:37:02.436171 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:37:02.436176 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:37:02.436321 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:37:02.436342 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 22:37:02.436439 (+ 97us) client_negotiation.cc:240] Negotiation successful
0810 22:37:02.436520 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"mutex_wait_us":338,"thread_start_us":163,"threads_started":1}
I20260810 22:37:02.437315 2565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84be25a3db6849fd99efa3182fb27460" candidate_uuid: "f1e19b4ab0404fbeb1208e06ed4bf0ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32bc0a72e0f7471186c4b04c2e8a0422" is_pre_election: true
I20260810 22:37:02.437383 2415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84be25a3db6849fd99efa3182fb27460" candidate_uuid: "f1e19b4ab0404fbeb1208e06ed4bf0ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7df0146ce9c40c9a2d24d9834a2125f" is_pre_election: true
I20260810 22:37:02.437662 2565 raft_consensus.cc:2454] T 84be25a3db6849fd99efa3182fb27460 P 32bc0a72e0f7471186c4b04c2e8a0422 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e19b4ab0404fbeb1208e06ed4bf0ee in term 0.
I20260810 22:37:02.437702 2415 raft_consensus.cc:2454] T 84be25a3db6849fd99efa3182fb27460 P f7df0146ce9c40c9a2d24d9834a2125f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e19b4ab0404fbeb1208e06ed4bf0ee in term 0.
I20260810 22:37:02.438185 2642 leader_election.cc:302] T 84be25a3db6849fd99efa3182fb27460 P f1e19b4ab0404fbeb1208e06ed4bf0ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32bc0a72e0f7471186c4b04c2e8a0422, f1e19b4ab0404fbeb1208e06ed4bf0ee; no voters:
I20260810 22:37:02.438493 2801 raft_consensus.cc:2781] T 84be25a3db6849fd99efa3182fb27460 P f1e19b4ab0404fbeb1208e06ed4bf0ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:37:02.438593 2801 raft_consensus.cc:482] T 84be25a3db6849fd99efa3182fb27460 P f1e19b4ab0404fbeb1208e06ed4bf0ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:37:02.438640 2801 raft_consensus.cc:3037] T 84be25a3db6849fd99efa3182fb27460 P f1e19b4ab0404fbeb1208e06ed4bf0ee [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:37:02.439706 2801 raft_consensus.cc:504] T 84be25a3db6849fd99efa3182fb27460 P f1e19b4ab0404fbeb1208e06ed4bf0ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7df0146ce9c40c9a2d24d9834a2125f" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 36229 } } peers { permanent_uuid: "32bc0a72e0f7471186c4b04c2e8a0422" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 34433 } } peers { permanent_uuid: "f1e19b4ab0404fbeb1208e06ed4bf0ee" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 38143 } }
I20260810 22:37:02.440088 2801 leader_election.cc:288] T 84be25a3db6849fd99efa3182fb27460 P f1e19b4ab0404fbeb1208e06ed4bf0ee [CANDIDATE]: Term 1 election: Requested vote from peers f7df0146ce9c40c9a2d24d9834a2125f (127.16.233.65:36229), 32bc0a72e0f7471186c4b04c2e8a0422 (127.16.233.66:34433)
I20260810 22:37:02.440491 2565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84be25a3db6849fd99efa3182fb27460" candidate_uuid: "f1e19b4ab0404fbeb1208e06ed4bf0ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32bc0a72e0f7471186c4b04c2e8a0422"
I20260810 22:37:02.440511 2415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84be25a3db6849fd99efa3182fb27460" candidate_uuid: "f1e19b4ab0404fbeb1208e06ed4bf0ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7df0146ce9c40c9a2d24d9834a2125f"
I20260810 22:37:02.440613 2565 raft_consensus.cc:3037] T 84be25a3db6849fd99efa3182fb27460 P 32bc0a72e0f7471186c4b04c2e8a0422 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:37:02.440712 2415 raft_consensus.cc:3037] T 84be25a3db6849fd99efa3182fb27460 P f7df0146ce9c40c9a2d24d9834a2125f [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:37:02.441793 2565 raft_consensus.cc:2454] T 84be25a3db6849fd99efa3182fb27460 P 32bc0a72e0f7471186c4b04c2e8a0422 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e19b4ab0404fbeb1208e06ed4bf0ee in term 1.
I20260810 22:37:02.442034 2415 raft_consensus.cc:2454] T 84be25a3db6849fd99efa3182fb27460 P f7df0146ce9c40c9a2d24d9834a2125f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e19b4ab0404fbeb1208e06ed4bf0ee in term 1.
I20260810 22:37:02.442190 2642 leader_election.cc:302] T 84be25a3db6849fd99efa3182fb27460 P f1e19b4ab0404fbeb1208e06ed4bf0ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32bc0a72e0f7471186c4b04c2e8a0422, f1e19b4ab0404fbeb1208e06ed4bf0ee; no voters:
I20260810 22:37:02.442389 2801 raft_consensus.cc:2781] T 84be25a3db6849fd99efa3182fb27460 P f1e19b4ab0404fbeb1208e06ed4bf0ee [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:37:02.442642 2801 raft_consensus.cc:686] T 84be25a3db6849fd99efa3182fb27460 P f1e19b4ab0404fbeb1208e06ed4bf0ee [term 1 LEADER]: Becoming Leader. State: Replica: f1e19b4ab0404fbeb1208e06ed4bf0ee, State: Running, Role: LEADER
I20260810 22:37:02.443058 2801 consensus_queue.cc:237] T 84be25a3db6849fd99efa3182fb27460 P f1e19b4ab0404fbeb1208e06ed4bf0ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7df0146ce9c40c9a2d24d9834a2125f" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 36229 } } peers { permanent_uuid: "32bc0a72e0f7471186c4b04c2e8a0422" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 34433 } } peers { permanent_uuid: "f1e19b4ab0404fbeb1208e06ed4bf0ee" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 38143 } }
I20260810 22:37:02.446372 2255 catalog_manager.cc:5495] T 84be25a3db6849fd99efa3182fb27460 P f1e19b4ab0404fbeb1208e06ed4bf0ee reported cstate change: term changed from 0 to 1, leader changed from <none> to f1e19b4ab0404fbeb1208e06ed4bf0ee (127.16.233.67). New cstate: current_term: 1 leader_uuid: "f1e19b4ab0404fbeb1208e06ed4bf0ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7df0146ce9c40c9a2d24d9834a2125f" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 36229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32bc0a72e0f7471186c4b04c2e8a0422" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 34433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1e19b4ab0404fbeb1208e06ed4bf0ee" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 38143 } health_report { overall_health: HEALTHY } } }
I20260810 22:37:02.473281 2791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:02.469554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37412 (local address 127.16.233.67:38143)
0810 22:37:02.469672 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 22:37:02.469675 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:37:02.469688 (+ 13us) server_negotiation.cc:407] Connection header received
0810 22:37:02.469808 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:37:02.469810 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:37:02.469868 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:37:02.469977 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:37:02.470207 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:37:02.470700 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:37:02.471647 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:37:02.471949 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:37:02.472027 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:37:02.472323 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:37:02.472796 (+ 473us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:37:02.473023 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0810 22:37:02.473088 (+ 65us) server_negotiation.cc:299] Negotiation successful
0810 22:37:02.473141 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 22:37:02.479815 2415 raft_consensus.cc:1261] T 84be25a3db6849fd99efa3182fb27460 P f7df0146ce9c40c9a2d24d9834a2125f [term 1 FOLLOWER]: Refusing update from remote peer f1e19b4ab0404fbeb1208e06ed4bf0ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:37:02.479888 2565 raft_consensus.cc:1261] T 84be25a3db6849fd99efa3182fb27460 P 32bc0a72e0f7471186c4b04c2e8a0422 [term 1 FOLLOWER]: Refusing update from remote peer f1e19b4ab0404fbeb1208e06ed4bf0ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:37:02.480647 2801 consensus_queue.cc:1035] T 84be25a3db6849fd99efa3182fb27460 P f1e19b4ab0404fbeb1208e06ed4bf0ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7df0146ce9c40c9a2d24d9834a2125f" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 36229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:37:02.480973 2804 consensus_queue.cc:1035] T 84be25a3db6849fd99efa3182fb27460 P f1e19b4ab0404fbeb1208e06ed4bf0ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "32bc0a72e0f7471186c4b04c2e8a0422" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 34433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:37:02.493511 2809 mvcc.cc:205] Tried to move back new op lower bound from 7317100226468917248 to 7317100226329436160. Current Snapshot: MvccSnapshot[applied={T|T < 7317100226468917248}]
I20260810 22:37:02.500579 2255 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40430:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:37:02.500771 2255 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40430:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:37:02.503605 2255 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c7109630f3354b35aa075a1f467262d4: Sending DeleteTablet for 3 replicas of tablet 84be25a3db6849fd99efa3182fb27460
I20260810 22:37:02.504554 2696 tablet_service.cc:1514] Processing DeleteTablet for tablet 84be25a3db6849fd99efa3182fb27460 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:37:02 UTC) from {username='kudu', principal='kudu/127.16.233.126@KRBTEST.COM'} at 127.0.0.1:37410
I20260810 22:37:02.504596 2541 tablet_service.cc:1514] Processing DeleteTablet for tablet 84be25a3db6849fd99efa3182fb27460 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:37:02 UTC) from {username='kudu', principal='kudu/127.16.233.126@KRBTEST.COM'} at 127.0.0.1:52216
I20260810 22:37:02.504696 2394 tablet_service.cc:1514] Processing DeleteTablet for tablet 84be25a3db6849fd99efa3182fb27460 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:37:02 UTC) from {username='kudu', principal='kudu/127.16.233.126@KRBTEST.COM'} at 127.0.0.1:39038
I20260810 22:37:02.504892 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 2324
I20260810 22:37:02.513154 2821 tablet_replica.cc:330] T 84be25a3db6849fd99efa3182fb27460 P 32bc0a72e0f7471186c4b04c2e8a0422: stopping tablet replica
I20260810 22:37:02.513622 2821 raft_consensus.cc:2229] T 84be25a3db6849fd99efa3182fb27460 P 32bc0a72e0f7471186c4b04c2e8a0422 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:37:02.513763 2820 tablet_replica.cc:330] T 84be25a3db6849fd99efa3182fb27460 P f1e19b4ab0404fbeb1208e06ed4bf0ee: stopping tablet replica
I20260810 22:37:02.514051 2821 raft_consensus.cc:2258] T 84be25a3db6849fd99efa3182fb27460 P 32bc0a72e0f7471186c4b04c2e8a0422 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:37:02.514082 2820 raft_consensus.cc:2229] T 84be25a3db6849fd99efa3182fb27460 P f1e19b4ab0404fbeb1208e06ed4bf0ee [term 1 LEADER]: Raft consensus shutting down.
I20260810 22:37:02.514726 2820 raft_consensus.cc:2258] T 84be25a3db6849fd99efa3182fb27460 P f1e19b4ab0404fbeb1208e06ed4bf0ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:37:02.515771 2821 ts_tablet_manager.cc:1905] T 84be25a3db6849fd99efa3182fb27460 P 32bc0a72e0f7471186c4b04c2e8a0422: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:37:02.516151 2820 ts_tablet_manager.cc:1905] T 84be25a3db6849fd99efa3182fb27460 P f1e19b4ab0404fbeb1208e06ed4bf0ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:37:02.518934 2821 ts_tablet_manager.cc:1918] T 84be25a3db6849fd99efa3182fb27460 P 32bc0a72e0f7471186c4b04c2e8a0422: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:37:02.518958 2820 ts_tablet_manager.cc:1918] T 84be25a3db6849fd99efa3182fb27460 P f1e19b4ab0404fbeb1208e06ed4bf0ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:37:02.519098 2821 log.cc:1196] T 84be25a3db6849fd99efa3182fb27460 P 32bc0a72e0f7471186c4b04c2e8a0422: Deleting WAL directory at /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-1/wal/wals/84be25a3db6849fd99efa3182fb27460
I20260810 22:37:02.519112 2820 log.cc:1196] T 84be25a3db6849fd99efa3182fb27460 P f1e19b4ab0404fbeb1208e06ed4bf0ee: Deleting WAL directory at /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-2/wal/wals/84be25a3db6849fd99efa3182fb27460
I20260810 22:37:02.519491 2821 ts_tablet_manager.cc:1939] T 84be25a3db6849fd99efa3182fb27460 P 32bc0a72e0f7471186c4b04c2e8a0422: Deleting consensus metadata
I20260810 22:37:02.519491 2820 ts_tablet_manager.cc:1939] T 84be25a3db6849fd99efa3182fb27460 P f1e19b4ab0404fbeb1208e06ed4bf0ee: Deleting consensus metadata
I20260810 22:37:02.520897 2238 catalog_manager.cc:4841] TS 32bc0a72e0f7471186c4b04c2e8a0422 (127.16.233.66:34433): tablet 84be25a3db6849fd99efa3182fb27460 (table test-table [id=c9f38f3c4541496b8a4e2975e0e7172b]) successfully deleted
I20260810 22:37:02.520897 2240 catalog_manager.cc:4841] TS f1e19b4ab0404fbeb1208e06ed4bf0ee (127.16.233.67:38143): tablet 84be25a3db6849fd99efa3182fb27460 (table test-table [id=c9f38f3c4541496b8a4e2975e0e7172b]) successfully deleted
I20260810 22:37:02.521907 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 2475
W20260810 22:37:02.521629 2240 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.233.65:36229 (error 108)
I20260810 22:37:02.525246 2786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:02.524833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.65:36229 (local address 127.0.0.1:39042)
0810 22:37:02.525055 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0810 22:37:02.525130 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.65:36229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
W20260810 22:37:02.525472 2240 catalog_manager.cc:4584] TS f7df0146ce9c40c9a2d24d9834a2125f (127.16.233.65:36229): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 84be25a3db6849fd99efa3182fb27460: Network error: Client connection negotiation failed: client connection to 127.16.233.65:36229: connect: Connection refused (error 111)
I20260810 22:37:02.534082 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 2623
I20260810 22:37:02.545980 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 2219
I20260810 22:37:22.555809 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:39115
--webserver_interface=127.16.233.126
--webserver_port=43295
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab
--principal=kudu/127.16.233.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:38849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.126:39115
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:37:22.681079 2823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:37:22.681380 2823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:37:22.681427 2823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:37:22.685735 2823 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 22:37:22.685798 2823 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 22:37:22.685815 2823 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 22:37:22.685837 2823 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:37:22.685855 2823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:37:22.685873 2823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:37:22.685941 2823 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:37:22.685962 2823 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:37:22.691555 2823 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:38849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.16.233.126:39115
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab
--principal=kudu/127.16.233.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:39115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.126
--webserver_port=43295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:37:22.692766 2823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:37:22.693935 2823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:37:22.709205 2829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:37:22.709322 2823 server_base.cc:1034] running on GCE node
W20260810 22:37:22.709205 2830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:37:22.709515 2832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:37:22.710148 2823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:37:22.711410 2823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:37:22.712580 2823 hybrid_clock.cc:648] HybridClock initialized: now 1786401442712547 us; error 49 us; skew 500 ppm
Aug 10 22:37:22 dist-test-slave-77v9 krb5kdc[2204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401442, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:37:22.723583 2823 init.cc:377] Logged in from keytab as kudu/127.16.233.126@KRBTEST.COM (short username kudu)
I20260810 22:37:22.724704 2823 webserver.cc:460] Webserver started at http://127.16.233.126:43295/ using document root <none> and password file <none>
I20260810 22:37:22.725397 2823 fs_manager.cc:359] Metadata directory not provided
I20260810 22:37:22.725483 2823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:37:22.730032 2823 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 22:37:22.732508 2840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:37:22.733880 2823 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 22:37:22.734046 2823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/master-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/master-0/wal
uuid: "c7109630f3354b35aa075a1f467262d4"
format_stamp: "Formatted at 2026-08-10 22:37:01 on dist-test-slave-77v9"
server_key: "2eff5433112c9d11164a106669587b59"
server_key_iv: "8d8791d9ad2768b560b36d7faa4cbe00"
server_key_version: "encryptionkey@0"
I20260810 22:37:22.734686 2823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:37:22.760406 2823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:37:22.764038 2823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:37:22.764217 2823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:37:22.772451 2823 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.126:39115
I20260810 22:37:22.772764 2897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.126:39115 every 8 connection(s)
I20260810 22:37:22.773788 2823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
I20260810 22:37:22.774662 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 2823
I20260810 22:37:22.775118 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.65:36229
--local_ip_for_outbound_sockets=127.16.233.65
--tserver_master_addrs=127.16.233.126:39115
--webserver_port=38839
--webserver_interface=127.16.233.65
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab
--principal=kudu/127.16.233.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:38849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 22:37:22.779722 2898 sys_catalog.cc:263] Verifying existing consensus state
I20260810 22:37:22.782877 2898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7109630f3354b35aa075a1f467262d4: Bootstrap starting.
I20260810 22:37:22.808094 2898 log.cc:824] T 00000000000000000000000000000000 P c7109630f3354b35aa075a1f467262d4: Log is configured to *not* fsync() on all Append() calls
I20260810 22:37:22.817955 2898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7109630f3354b35aa075a1f467262d4: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 22:37:22.818439 2898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7109630f3354b35aa075a1f467262d4: Bootstrap complete.
I20260810 22:37:22.823247 2898 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7109630f3354b35aa075a1f467262d4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7109630f3354b35aa075a1f467262d4" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39115 } }
I20260810 22:37:22.823706 2898 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7109630f3354b35aa075a1f467262d4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7109630f3354b35aa075a1f467262d4, State: Initialized, Role: FOLLOWER
I20260810 22:37:22.824321 2898 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7109630f3354b35aa075a1f467262d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7109630f3354b35aa075a1f467262d4" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39115 } }
I20260810 22:37:22.824489 2898 raft_consensus.cc:388] T 00000000000000000000000000000000 P c7109630f3354b35aa075a1f467262d4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:37:22.824575 2898 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7109630f3354b35aa075a1f467262d4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:37:22.824698 2898 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7109630f3354b35aa075a1f467262d4 [term 1 FOLLOWER]: Advancing to term 2
I20260810 22:37:22.826906 2898 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7109630f3354b35aa075a1f467262d4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7109630f3354b35aa075a1f467262d4" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39115 } }
I20260810 22:37:22.827291 2898 leader_election.cc:302] T 00000000000000000000000000000000 P c7109630f3354b35aa075a1f467262d4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c7109630f3354b35aa075a1f467262d4; no voters:
I20260810 22:37:22.827677 2898 leader_election.cc:288] T 00000000000000000000000000000000 P c7109630f3354b35aa075a1f467262d4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 22:37:22.827809 2902 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7109630f3354b35aa075a1f467262d4 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 22:37:22.828071 2902 raft_consensus.cc:686] T 00000000000000000000000000000000 P c7109630f3354b35aa075a1f467262d4 [term 2 LEADER]: Becoming Leader. State: Replica: c7109630f3354b35aa075a1f467262d4, State: Running, Role: LEADER
I20260810 22:37:22.828398 2902 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7109630f3354b35aa075a1f467262d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7109630f3354b35aa075a1f467262d4" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39115 } }
I20260810 22:37:22.829129 2898 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7109630f3354b35aa075a1f467262d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:37:22.829962 2903 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7109630f3354b35aa075a1f467262d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c7109630f3354b35aa075a1f467262d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7109630f3354b35aa075a1f467262d4" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39115 } } }
I20260810 22:37:22.830122 2903 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7109630f3354b35aa075a1f467262d4 [sys.catalog]: This master's current role is: LEADER
I20260810 22:37:22.830892 2912 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:37:22.830847 2904 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7109630f3354b35aa075a1f467262d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7109630f3354b35aa075a1f467262d4. Latest consensus state: current_term: 2 leader_uuid: "c7109630f3354b35aa075a1f467262d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7109630f3354b35aa075a1f467262d4" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39115 } } }
I20260810 22:37:22.831080 2904 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7109630f3354b35aa075a1f467262d4 [sys.catalog]: This master's current role is: LEADER
I20260810 22:37:22.835927 2912 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:37:22.836834 2912 catalog_manager.cc:1259] Loaded cluster ID: b3ab420c4a7442a2bc0fcda8f9d00e2b
I20260810 22:37:22.836897 2912 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:37:22.839326 2912 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:37:22.839986 2912 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c7109630f3354b35aa075a1f467262d4: Loaded TSK: 0
I20260810 22:37:22.852576 2912 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c7109630f3354b35aa075a1f467262d4: Generated new TSK 1
I20260810 22:37:22.852898 2912 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 22:37:22.909142 2900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:37:22.909425 2900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:37:22.909509 2900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:37:22.913687 2900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:37:22.913766 2900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:37:22.913870 2900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.65
I20260810 22:37:22.919574 2900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:38849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab
--principal=kudu/127.16.233.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.65:36229
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.233.65
--webserver_port=38839
--tserver_master_addrs=127.16.233.126:39115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.65
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:37:22.920905 2900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:37:22.922050 2900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:37:22.933924 2930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:37:22.933925 2928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:37:22.933907 2927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:37:22.934517 2900 server_base.cc:1034] running on GCE node
I20260810 22:37:22.934996 2900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:37:22.935715 2900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:37:22.936905 2900 hybrid_clock.cc:648] HybridClock initialized: now 1786401442936892 us; error 34 us; skew 500 ppm
Aug 10 22:37:22 dist-test-slave-77v9 krb5kdc[2204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401442, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:37:22.940160 2900 init.cc:377] Logged in from keytab as kudu/127.16.233.65@KRBTEST.COM (short username kudu)
I20260810 22:37:22.941313 2900 webserver.cc:460] Webserver started at http://127.16.233.65:38839/ using document root <none> and password file <none>
I20260810 22:37:22.941910 2900 fs_manager.cc:359] Metadata directory not provided
I20260810 22:37:22.942025 2900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:37:22.946328 2900 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 22:37:22.948882 2938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:37:22.950321 2900 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:37:22.950487 2900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-0/wal
uuid: "f7df0146ce9c40c9a2d24d9834a2125f"
format_stamp: "Formatted at 2026-08-10 22:37:01 on dist-test-slave-77v9"
server_key: "b284566687ddcb1a6eca3bfc282fa8db"
server_key_iv: "f6cb8a8b6cbd9908e77830f03c31c8c3"
server_key_version: "encryptionkey@0"
I20260810 22:37:22.951146 2900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:37:22.991230 2900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:37:22.994334 2900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:37:22.994562 2900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:37:22.995186 2900 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:37:22.996289 2945 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 22:37:23.000528 2900 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 22:37:23.000622 2900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260810 22:37:23.000705 2900 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 22:37:23.002446 2900 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 22:37:23.002524 2900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260810 22:37:23.002849 2945 tablet_bootstrap.cc:492] T 84be25a3db6849fd99efa3182fb27460 P f7df0146ce9c40c9a2d24d9834a2125f: Bootstrap starting.
I20260810 22:37:23.013762 2900 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.65:36229
I20260810 22:37:23.013846 3060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.65:36229 every 8 connection(s)
I20260810 22:37:23.014776 2900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
I20260810 22:37:23.016702 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 2900
I20260810 22:37:23.017256 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.66:34433
--local_ip_for_outbound_sockets=127.16.233.66
--tserver_master_addrs=127.16.233.126:39115
--webserver_port=41565
--webserver_interface=127.16.233.66
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab
--principal=kudu/127.16.233.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:38849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 22:37:23.023272 2945 log.cc:824] T 84be25a3db6849fd99efa3182fb27460 P f7df0146ce9c40c9a2d24d9834a2125f: Log is configured to *not* fsync() on all Append() calls
Aug 10 22:37:23 dist-test-slave-77v9 krb5kdc[2204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401442, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.65@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:37:23.032145 2945 tablet_bootstrap.cc:492] T 84be25a3db6849fd99efa3182fb27460 P f7df0146ce9c40c9a2d24d9834a2125f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 22:37:23.032651 2945 tablet_bootstrap.cc:492] T 84be25a3db6849fd99efa3182fb27460 P f7df0146ce9c40c9a2d24d9834a2125f: Bootstrap complete.
I20260810 22:37:23.034261 2945 ts_tablet_manager.cc:1397] T 84be25a3db6849fd99efa3182fb27460 P f7df0146ce9c40c9a2d24d9834a2125f: Time spent bootstrapping tablet: real 0.032s user 0.026s sys 0.004s
I20260810 22:37:23.036980 3063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:23.016921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:40235 (local address 127.16.233.126:39115)
0810 22:37:23.017768 (+ 847us) server_negotiation.cc:206] Beginning negotiation
0810 22:37:23.017796 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0810 22:37:23.018413 (+ 617us) server_negotiation.cc:407] Connection header received
0810 22:37:23.020015 (+ 1602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:37:23.020033 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:37:23.020468 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:37:23.020770 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:37:23.022363 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:37:23.024082 (+ 1719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:37:23.025895 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:37:23.026765 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:37:23.026980 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:37:23.030624 (+ 3644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:37:23.030631 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:37:23.030647 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:37:23.030684 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:37:23.033203 (+ 2519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:37:23.033872 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:37:23.033881 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:37:23.033890 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:37:23.034068 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:37:23.034477 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:37:23.034483 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:37:23.034486 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:37:23.034962 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:37:23.035094 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 22:37:23.035767 (+ 673us) server_negotiation.cc:299] Negotiation successful
0810 22:37:23.036084 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":138,"threads_started":1}
I20260810 22:37:23.036988 3065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:23.016908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39115 (local address 127.16.233.65:40235)
0810 22:37:23.018172 (+ 1264us) negotiation.cc:107] Waiting for socket to connect
0810 22:37:23.018243 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0810 22:37:23.019477 (+ 1234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:37:23.020962 (+ 1485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:37:23.020975 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:37:23.021493 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:37:23.022150 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:37:23.022172 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:37:23.024329 (+ 2157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:37:23.024336 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:37:23.025740 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:37:23.025751 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:37:23.026973 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:37:23.026977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:37:23.027168 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:37:23.027933 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:37:23.027953 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:37:23.030313 (+ 2360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:37:23.033352 (+ 3039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:37:23.033358 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:37:23.033372 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:37:23.033726 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:37:23.034178 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:37:23.034181 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:37:23.034183 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:37:23.034356 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:37:23.035097 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:37:23.035104 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:37:23.035415 (+ 311us) client_negotiation.cc:769] Sending connection context
0810 22:37:23.035612 (+ 197us) client_negotiation.cc:240] Negotiation successful
0810 22:37:23.035960 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":986,"thread_start_us":167,"threads_started":1}
I20260810 22:37:23.037246 2945 raft_consensus.cc:348] T 84be25a3db6849fd99efa3182fb27460 P f7df0146ce9c40c9a2d24d9834a2125f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7df0146ce9c40c9a2d24d9834a2125f" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 36229 } } peers { permanent_uuid: "32bc0a72e0f7471186c4b04c2e8a0422" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 34433 } } peers { permanent_uuid: "f1e19b4ab0404fbeb1208e06ed4bf0ee" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 38143 } }
I20260810 22:37:23.037757 2945 raft_consensus.cc:729] T 84be25a3db6849fd99efa3182fb27460 P f7df0146ce9c40c9a2d24d9834a2125f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7df0146ce9c40c9a2d24d9834a2125f, State: Initialized, Role: FOLLOWER
I20260810 22:37:23.038381 2945 consensus_queue.cc:260] T 84be25a3db6849fd99efa3182fb27460 P f7df0146ce9c40c9a2d24d9834a2125f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7df0146ce9c40c9a2d24d9834a2125f" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 36229 } } peers { permanent_uuid: "32bc0a72e0f7471186c4b04c2e8a0422" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 34433 } } peers { permanent_uuid: "f1e19b4ab0404fbeb1208e06ed4bf0ee" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 38143 } }
I20260810 22:37:23.040025 3061 heartbeater.cc:347] Connected to a master server at 127.16.233.126:39115
I20260810 22:37:23.040243 2945 ts_tablet_manager.cc:1428] T 84be25a3db6849fd99efa3182fb27460 P f7df0146ce9c40c9a2d24d9834a2125f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 22:37:23.040272 3061 heartbeater.cc:464] Registering TS with master...
I20260810 22:37:23.040864 3061 heartbeater.cc:511] Master 127.16.233.126:39115 requested a full tablet report, sending...
I20260810 22:37:23.043315 2858 ts_manager.cc:194] Registered new tserver with Master: f7df0146ce9c40c9a2d24d9834a2125f (127.16.233.65:36229)
I20260810 22:37:23.047186 2858 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.65@KRBTEST.COM'} at 127.16.233.65:40235
I20260810 22:37:23.050105 3061 heartbeater.cc:503] Master 127.16.233.126:39115 was elected leader, sending a full tablet report...
Aug 10 22:37:23 dist-test-slave-77v9 krb5kdc[2204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401442, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.126@KRBTEST.COM for kudu/127.16.233.65@KRBTEST.COM
I20260810 22:37:23.057389 3072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:23.046211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39098 (local address 127.16.233.65:36229)
0810 22:37:23.046895 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0810 22:37:23.046905 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 22:37:23.046928 (+ 23us) server_negotiation.cc:407] Connection header received
0810 22:37:23.046991 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:37:23.046996 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:37:23.047182 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:37:23.047350 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:37:23.048184 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:37:23.049051 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:37:23.049987 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:37:23.050413 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:37:23.050532 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:37:23.053222 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:37:23.053235 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:37:23.053250 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:37:23.053289 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:37:23.054943 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:37:23.055650 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:37:23.055658 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:37:23.055660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:37:23.055766 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:37:23.056063 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:37:23.056081 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:37:23.056083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:37:23.056467 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:37:23.056621 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0810 22:37:23.057052 (+ 431us) server_negotiation.cc:299] Negotiation successful
0810 22:37:23.057227 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":589,"thread_start_us":112,"threads_started":1}
I20260810 22:37:23.057387 3071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:23.046099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.65:36229 (local address 127.0.0.1:39098)
0810 22:37:23.046557 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0810 22:37:23.046609 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 22:37:23.046741 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:37:23.047459 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:37:23.047465 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:37:23.047922 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:37:23.048053 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:37:23.048062 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:37:23.049191 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:37:23.049195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:37:23.049852 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:37:23.049863 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:37:23.050550 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:37:23.050555 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:37:23.050679 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:37:23.051335 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:37:23.051349 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:37:23.053052 (+ 1703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:37:23.055132 (+ 2080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:37:23.055142 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:37:23.055165 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:37:23.055526 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:37:23.055868 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:37:23.055871 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:37:23.055873 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:37:23.055967 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:37:23.056594 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:37:23.056600 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:37:23.056735 (+ 135us) client_negotiation.cc:769] Sending connection context
0810 22:37:23.056965 (+ 230us) client_negotiation.cc:240] Negotiation successful
0810 22:37:23.057190 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":141,"threads_started":1}
I20260810 22:37:23.058921 2991 tablet_service.cc:1514] Processing DeleteTablet for tablet 84be25a3db6849fd99efa3182fb27460 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:37:02 UTC) from {username='kudu', principal='kudu/127.16.233.126@KRBTEST.COM'} at 127.0.0.1:39098
I20260810 22:37:23.060053 3073 tablet_replica.cc:330] stopping tablet replica
I20260810 22:37:23.060503 3073 raft_consensus.cc:2229] T 84be25a3db6849fd99efa3182fb27460 P f7df0146ce9c40c9a2d24d9834a2125f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:37:23.060801 3073 raft_consensus.cc:2258] T 84be25a3db6849fd99efa3182fb27460 P f7df0146ce9c40c9a2d24d9834a2125f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:37:23.061937 3073 ts_tablet_manager.cc:1905] T 84be25a3db6849fd99efa3182fb27460 P f7df0146ce9c40c9a2d24d9834a2125f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:37:23.066323 3073 ts_tablet_manager.cc:1918] T 84be25a3db6849fd99efa3182fb27460 P f7df0146ce9c40c9a2d24d9834a2125f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:37:23.066481 3073 log.cc:1196] T 84be25a3db6849fd99efa3182fb27460 P f7df0146ce9c40c9a2d24d9834a2125f: Deleting WAL directory at /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-0/wal/wals/84be25a3db6849fd99efa3182fb27460
I20260810 22:37:23.066800 3073 ts_tablet_manager.cc:1939] T 84be25a3db6849fd99efa3182fb27460 P f7df0146ce9c40c9a2d24d9834a2125f: Deleting consensus metadata
I20260810 22:37:23.068682 2844 catalog_manager.cc:4841] TS f7df0146ce9c40c9a2d24d9834a2125f (127.16.233.65:36229): tablet 84be25a3db6849fd99efa3182fb27460 (table test-table [id=c9f38f3c4541496b8a4e2975e0e7172b]) successfully deleted
W20260810 22:37:23.142732 3067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:37:23.143010 3067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:37:23.143056 3067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:37:23.147099 3067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:37:23.147166 3067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:37:23.147228 3067 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.66
I20260810 22:37:23.152740 3067 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:38849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab
--principal=kudu/127.16.233.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.66:34433
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.233.66
--webserver_port=41565
--tserver_master_addrs=127.16.233.126:39115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3067
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.66
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:37:23.153891 3067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:37:23.154985 3067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:37:23.165297 3078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:37:23.165292 3081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:37:23.165526 3067 server_base.cc:1034] running on GCE node
W20260810 22:37:23.165333 3079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:37:23.166167 3067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:37:23.166949 3067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:37:23.168125 3067 hybrid_clock.cc:648] HybridClock initialized: now 1786401443168093 us; error 41 us; skew 500 ppm
Aug 10 22:37:23 dist-test-slave-77v9 krb5kdc[2204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401443, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:37:23.171097 3067 init.cc:377] Logged in from keytab as kudu/127.16.233.66@KRBTEST.COM (short username kudu)
I20260810 22:37:23.171993 3067 webserver.cc:460] Webserver started at http://127.16.233.66:41565/ using document root <none> and password file <none>
I20260810 22:37:23.172569 3067 fs_manager.cc:359] Metadata directory not provided
I20260810 22:37:23.172684 3067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:37:23.176687 3067 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260810 22:37:23.178949 3089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:37:23.180209 3067 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:37:23.180349 3067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-1/wal
uuid: "32bc0a72e0f7471186c4b04c2e8a0422"
format_stamp: "Formatted at 2026-08-10 22:37:02 on dist-test-slave-77v9"
server_key: "90aac31c1a5f558d2a48a0697537b93a"
server_key_iv: "f08935cd48ca5afccf11d200362e9c96"
server_key_version: "encryptionkey@0"
I20260810 22:37:23.180991 3067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:37:23.210795 3067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:37:23.214515 3067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:37:23.214773 3067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:37:23.215461 3067 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:37:23.216383 3067 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:37:23.216471 3067 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:37:23.216557 3067 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:37:23.216604 3067 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:37:23.227314 3067 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.66:34433
I20260810 22:37:23.227388 3217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.66:34433 every 8 connection(s)
I20260810 22:37:23.228487 3067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
I20260810 22:37:23.237185 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 3067
I20260810 22:37:23.237751 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.67:38143
--local_ip_for_outbound_sockets=127.16.233.67
--tserver_master_addrs=127.16.233.126:39115
--webserver_port=35907
--webserver_interface=127.16.233.67
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab
--principal=kudu/127.16.233.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:38849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 22:37:23 dist-test-slave-77v9 krb5kdc[2204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401443, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.66@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:37:23.244992 3063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:23.230828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:54119 (local address 127.16.233.126:39115)
0810 22:37:23.231041 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0810 22:37:23.231047 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:37:23.232005 (+ 958us) server_negotiation.cc:407] Connection header received
0810 22:37:23.233660 (+ 1655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:37:23.233663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:37:23.233749 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:37:23.233832 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:37:23.235142 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:37:23.235675 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:37:23.237053 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:37:23.237280 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:37:23.237431 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:37:23.240657 (+ 3226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:37:23.240661 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:37:23.240663 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:37:23.240690 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:37:23.242752 (+ 2062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:37:23.243308 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:37:23.243315 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:37:23.243320 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:37:23.243431 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:37:23.243782 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:37:23.243786 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:37:23.243787 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:37:23.243994 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:37:23.244111 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 22:37:23.244635 (+ 524us) server_negotiation.cc:299] Negotiation successful
0810 22:37:23.244779 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 22:37:23.245814 3220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:23.231290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39115 (local address 127.16.233.66:54119)
0810 22:37:23.231802 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0810 22:37:23.231850 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 22:37:23.233355 (+ 1505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:37:23.234016 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:37:23.234022 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:37:23.234454 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:37:23.234993 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:37:23.235007 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:37:23.235829 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:37:23.235832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:37:23.236922 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:37:23.236938 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:37:23.237506 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:37:23.237512 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:37:23.237692 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:37:23.238385 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:37:23.238409 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:37:23.240444 (+ 2035us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:37:23.242902 (+ 2458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:37:23.242907 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:37:23.242920 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:37:23.243187 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:37:23.243535 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:37:23.243537 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:37:23.243539 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:37:23.243675 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:37:23.244113 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:37:23.244122 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:37:23.244387 (+ 265us) client_negotiation.cc:769] Sending connection context
0810 22:37:23.244579 (+ 192us) client_negotiation.cc:240] Negotiation successful
0810 22:37:23.244953 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":122,"threads_started":1}
I20260810 22:37:23.247092 3218 heartbeater.cc:347] Connected to a master server at 127.16.233.126:39115
I20260810 22:37:23.247417 3218 heartbeater.cc:464] Registering TS with master...
I20260810 22:37:23.248162 3218 heartbeater.cc:511] Master 127.16.233.126:39115 requested a full tablet report, sending...
I20260810 22:37:23.249547 2858 ts_manager.cc:194] Registered new tserver with Master: 32bc0a72e0f7471186c4b04c2e8a0422 (127.16.233.66:34433)
I20260810 22:37:23.250555 2858 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.66@KRBTEST.COM'} at 127.16.233.66:54119
W20260810 22:37:23.365911 3221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:37:23.366233 3221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:37:23.366279 3221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:37:23.370589 3221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:37:23.370651 3221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:37:23.370713 3221 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.67
I20260810 22:37:23.376171 3221 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:38849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab
--principal=kudu/127.16.233.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.67:38143
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.233.67
--webserver_port=35907
--tserver_master_addrs=127.16.233.126:39115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3221
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.67
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:37:23.377322 3221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:37:23.378392 3221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:37:23.388564 3229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:37:23.388549 3228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:37:23.388542 3232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:37:23.388592 3221 server_base.cc:1034] running on GCE node
I20260810 22:37:23.389223 3221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:37:23.389969 3221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:37:23.391196 3221 hybrid_clock.cc:648] HybridClock initialized: now 1786401443391173 us; error 34 us; skew 500 ppm
Aug 10 22:37:23 dist-test-slave-77v9 krb5kdc[2204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401443, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:37:23.394476 3221 init.cc:377] Logged in from keytab as kudu/127.16.233.67@KRBTEST.COM (short username kudu)
I20260810 22:37:23.395506 3221 webserver.cc:460] Webserver started at http://127.16.233.67:35907/ using document root <none> and password file <none>
I20260810 22:37:23.396107 3221 fs_manager.cc:359] Metadata directory not provided
I20260810 22:37:23.396188 3221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:37:23.400552 3221 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:37:23.402921 3239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:37:23.404296 3221 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 22:37:23.404413 3221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-2/wal
uuid: "f1e19b4ab0404fbeb1208e06ed4bf0ee"
format_stamp: "Formatted at 2026-08-10 22:37:02 on dist-test-slave-77v9"
server_key: "b8e0bfe2c8721b4377d9f951121fd9d6"
server_key_iv: "62ce57aae310ed966623d263463c8469"
server_key_version: "encryptionkey@0"
I20260810 22:37:23.405145 3221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:37:23.419152 3221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:37:23.422531 3221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:37:23.422710 3221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:37:23.423338 3221 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:37:23.424239 3221 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:37:23.424291 3221 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:37:23.424340 3221 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:37:23.424365 3221 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:37:23.438000 3221 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.67:38143
I20260810 22:37:23.438103 3365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.67:38143 every 8 connection(s)
I20260810 22:37:23.439354 3221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
I20260810 22:37:23.447522 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 3221
Aug 10 22:37:23 dist-test-slave-77v9 krb5kdc[2204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401443, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.67@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:37:23.455546 3063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:23.441453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:52281 (local address 127.16.233.126:39115)
0810 22:37:23.441603 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 22:37:23.441607 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:37:23.442601 (+ 994us) server_negotiation.cc:407] Connection header received
0810 22:37:23.444001 (+ 1400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:37:23.444004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:37:23.444066 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:37:23.444181 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:37:23.445450 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:37:23.445992 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:37:23.447167 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:37:23.447426 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:37:23.447546 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:37:23.450814 (+ 3268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:37:23.450818 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:37:23.450820 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:37:23.450843 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:37:23.452526 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:37:23.453285 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:37:23.453290 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:37:23.453292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:37:23.453414 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:37:23.453857 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:37:23.453864 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:37:23.453869 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:37:23.454163 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:37:23.454285 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0810 22:37:23.455162 (+ 877us) server_negotiation.cc:299] Negotiation successful
0810 22:37:23.455307 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 22:37:23.456707 3368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:23.441962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39115 (local address 127.16.233.67:52281)
0810 22:37:23.442446 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0810 22:37:23.442490 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 22:37:23.443777 (+ 1287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:37:23.444357 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:37:23.444369 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:37:23.444752 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:37:23.445295 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:37:23.445308 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:37:23.446133 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:37:23.446136 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:37:23.447032 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:37:23.447042 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:37:23.447595 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:37:23.447599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:37:23.447821 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:37:23.448627 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:37:23.448657 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:37:23.450565 (+ 1908us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:37:23.452666 (+ 2101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:37:23.452672 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:37:23.452685 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:37:23.453153 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:37:23.453539 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:37:23.453541 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:37:23.453543 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:37:23.453688 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:37:23.454306 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:37:23.454312 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:37:23.454727 (+ 415us) client_negotiation.cc:769] Sending connection context
0810 22:37:23.454971 (+ 244us) client_negotiation.cc:240] Negotiation successful
0810 22:37:23.455393 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":128,"threads_started":1}
I20260810 22:37:23.458082 3366 heartbeater.cc:347] Connected to a master server at 127.16.233.126:39115
I20260810 22:37:23.458397 3366 heartbeater.cc:464] Registering TS with master...
I20260810 22:37:23.458487 3370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:23.448513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50544 (local address 127.16.233.126:39115)
0810 22:37:23.449205 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0810 22:37:23.449210 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:37:23.449222 (+ 12us) server_negotiation.cc:407] Connection header received
0810 22:37:23.449258 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:37:23.449260 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:37:23.449301 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:37:23.449409 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:37:23.450199 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:37:23.450930 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:37:23.452208 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:37:23.452445 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:37:23.452530 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:37:23.454468 (+ 1938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:37:23.454472 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:37:23.454474 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:37:23.454499 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:37:23.456450 (+ 1951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:37:23.457246 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:37:23.457249 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:37:23.457250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:37:23.457325 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:37:23.457633 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:37:23.457642 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:37:23.457644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:37:23.457809 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:37:23.458032 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0810 22:37:23.458171 (+ 139us) server_negotiation.cc:299] Negotiation successful
0810 22:37:23.458290 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":119,"threads_started":1}
I20260810 22:37:23.458999 3366 heartbeater.cc:511] Master 127.16.233.126:39115 requested a full tablet report, sending...
I20260810 22:37:23.460136 2857 ts_manager.cc:194] Registered new tserver with Master: f1e19b4ab0404fbeb1208e06ed4bf0ee (127.16.233.67:38143)
I20260810 22:37:23.460925 2857 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.67@KRBTEST.COM'} at 127.16.233.67:52281
I20260810 22:37:23.461338 17317 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 22:37:23.465521 3370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:23.462087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50550 (local address 127.16.233.126:39115)
0810 22:37:23.462220 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 22:37:23.462224 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:37:23.462416 (+ 192us) server_negotiation.cc:407] Connection header received
0810 22:37:23.462537 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:37:23.462540 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:37:23.462651 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:37:23.462759 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:37:23.463039 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:37:23.463613 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:37:23.464456 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:37:23.464672 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:37:23.464747 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:37:23.465097 (+ 350us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:37:23.465217 (+ 120us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0810 22:37:23.465421 (+ 204us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50550: token expired
Metrics: {"server-negotiator.queue_time_us":54}
W20260810 22:37:23.465608 3370 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50550: token expired
W20260810 22:37:23.466106 2774 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.16.233.126:39115)
I20260810 22:37:23.473851 3370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:23.466537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50554 (local address 127.16.233.126:39115)
0810 22:37:23.466657 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0810 22:37:23.466662 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:37:23.466701 (+ 39us) server_negotiation.cc:407] Connection header received
0810 22:37:23.466832 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:37:23.466837 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:37:23.466882 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:37:23.466970 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:37:23.467586 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:37:23.468071 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:37:23.468901 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:37:23.469110 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:37:23.469216 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:37:23.470616 (+ 1400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:37:23.470625 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:37:23.470627 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:37:23.470650 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:37:23.472104 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:37:23.472551 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:37:23.472559 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:37:23.472560 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:37:23.472636 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:37:23.472986 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:37:23.472991 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:37:23.472992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:37:23.473172 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:37:23.473318 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 22:37:23.473553 (+ 235us) server_negotiation.cc:299] Negotiation successful
0810 22:37:23.473675 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 22:37:23.476612 2857 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50554:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 22:37:23.479981 2857 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:37:23.488510 2991 tablet_service.cc:1467] Processing CreateTablet for tablet 9a2cb51d98444688a200c5d0278fcbb5 (DEFAULT_TABLE table=test-table [id=569bb4ae1f50461e8b0f557e424dfda5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:37:23.489249 2991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a2cb51d98444688a200c5d0278fcbb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:37:23.492297 2945 tablet_bootstrap.cc:492] T 9a2cb51d98444688a200c5d0278fcbb5 P f7df0146ce9c40c9a2d24d9834a2125f: Bootstrap starting.
I20260810 22:37:23.493376 2945 tablet_bootstrap.cc:654] T 9a2cb51d98444688a200c5d0278fcbb5 P f7df0146ce9c40c9a2d24d9834a2125f: Neither blocks nor log segments found. Creating new log.
I20260810 22:37:23.493974 3378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:23.488407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.66:34433 (local address 127.0.0.1:48394)
0810 22:37:23.488690 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0810 22:37:23.488706 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 22:37:23.488803 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:37:23.489552 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:37:23.489555 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:37:23.489574 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:37:23.489669 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:37:23.489676 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:37:23.491182 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:37:23.491185 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:37:23.492317 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:37:23.492328 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:37:23.493397 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:37:23.493401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:37:23.493513 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:37:23.493581 (+ 68us) client_negotiation.cc:769] Sending connection context
0810 22:37:23.493680 (+ 99us) client_negotiation.cc:240] Negotiation successful
0810 22:37:23.493783 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":1,"thread_start_us":93,"threads_started":1}
I20260810 22:37:23.494032 3379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:23.488588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48394 (local address 127.16.233.66:34433)
0810 22:37:23.489106 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0810 22:37:23.489118 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 22:37:23.489139 (+ 21us) server_negotiation.cc:407] Connection header received
0810 22:37:23.489214 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:37:23.489219 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:37:23.489396 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:37:23.489553 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:37:23.489873 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:37:23.491051 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:37:23.492460 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:37:23.493256 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:37:23.493371 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:37:23.493534 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0810 22:37:23.493731 (+ 197us) server_negotiation.cc:299] Negotiation successful
0810 22:37:23.493840 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":124,"threads_started":1}
I20260810 22:37:23.494176 3071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:23.487474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.67:38143 (local address 127.0.0.1:45806)
0810 22:37:23.487617 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0810 22:37:23.487632 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:37:23.487714 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:37:23.488830 (+ 1116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:37:23.488833 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:37:23.488847 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:37:23.488969 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:37:23.488977 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:37:23.490826 (+ 1849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:37:23.490829 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:37:23.492092 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:37:23.492103 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:37:23.493742 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:37:23.493745 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:37:23.493858 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:37:23.493881 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 22:37:23.493957 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 22:37:23.494048 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":35}
I20260810 22:37:23.494928 2945 tablet_bootstrap.cc:492] T 9a2cb51d98444688a200c5d0278fcbb5 P f7df0146ce9c40c9a2d24d9834a2125f: No bootstrap required, opened a new log
I20260810 22:37:23.495045 2945 ts_tablet_manager.cc:1397] T 9a2cb51d98444688a200c5d0278fcbb5 P f7df0146ce9c40c9a2d24d9834a2125f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 22:37:23.495519 2945 raft_consensus.cc:348] T 9a2cb51d98444688a200c5d0278fcbb5 P f7df0146ce9c40c9a2d24d9834a2125f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e19b4ab0404fbeb1208e06ed4bf0ee" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 38143 } } peers { permanent_uuid: "32bc0a72e0f7471186c4b04c2e8a0422" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 34433 } } peers { permanent_uuid: "f7df0146ce9c40c9a2d24d9834a2125f" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 36229 } }
I20260810 22:37:23.495644 2945 raft_consensus.cc:374] T 9a2cb51d98444688a200c5d0278fcbb5 P f7df0146ce9c40c9a2d24d9834a2125f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:37:23.495682 2945 raft_consensus.cc:729] T 9a2cb51d98444688a200c5d0278fcbb5 P f7df0146ce9c40c9a2d24d9834a2125f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7df0146ce9c40c9a2d24d9834a2125f, State: Initialized, Role: FOLLOWER
I20260810 22:37:23.495836 2945 consensus_queue.cc:260] T 9a2cb51d98444688a200c5d0278fcbb5 P f7df0146ce9c40c9a2d24d9834a2125f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e19b4ab0404fbeb1208e06ed4bf0ee" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 38143 } } peers { permanent_uuid: "32bc0a72e0f7471186c4b04c2e8a0422" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 34433 } } peers { permanent_uuid: "f7df0146ce9c40c9a2d24d9834a2125f" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 36229 } }
I20260810 22:37:23.495929 3377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:23.487907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45806 (local address 127.16.233.67:38143)
0810 22:37:23.488246 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0810 22:37:23.488260 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 22:37:23.488294 (+ 34us) server_negotiation.cc:407] Connection header received
0810 22:37:23.488382 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:37:23.488389 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:37:23.488627 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:37:23.489131 (+ 504us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:37:23.489327 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:37:23.490691 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:37:23.492251 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:37:23.493250 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:37:23.493371 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:37:23.493629 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0810 22:37:23.495651 (+ 2022us) server_negotiation.cc:299] Negotiation successful
0810 22:37:23.495765 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":141,"threads_started":1}
I20260810 22:37:23.496557 2945 ts_tablet_manager.cc:1428] T 9a2cb51d98444688a200c5d0278fcbb5 P f7df0146ce9c40c9a2d24d9834a2125f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 22:37:23.497973 3146 tablet_service.cc:1467] Processing CreateTablet for tablet 9a2cb51d98444688a200c5d0278fcbb5 (DEFAULT_TABLE table=test-table [id=569bb4ae1f50461e8b0f557e424dfda5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:37:23.498579 3282 tablet_service.cc:1467] Processing CreateTablet for tablet 9a2cb51d98444688a200c5d0278fcbb5 (DEFAULT_TABLE table=test-table [id=569bb4ae1f50461e8b0f557e424dfda5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:37:23.499156 3146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a2cb51d98444688a200c5d0278fcbb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:37:23.499619 3282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a2cb51d98444688a200c5d0278fcbb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:37:23.505947 3384 tablet_bootstrap.cc:492] T 9a2cb51d98444688a200c5d0278fcbb5 P f1e19b4ab0404fbeb1208e06ed4bf0ee: Bootstrap starting.
I20260810 22:37:23.506376 3382 tablet_bootstrap.cc:492] T 9a2cb51d98444688a200c5d0278fcbb5 P 32bc0a72e0f7471186c4b04c2e8a0422: Bootstrap starting.
I20260810 22:37:23.508414 3382 tablet_bootstrap.cc:654] T 9a2cb51d98444688a200c5d0278fcbb5 P 32bc0a72e0f7471186c4b04c2e8a0422: Neither blocks nor log segments found. Creating new log.
I20260810 22:37:23.508428 3384 tablet_bootstrap.cc:654] T 9a2cb51d98444688a200c5d0278fcbb5 P f1e19b4ab0404fbeb1208e06ed4bf0ee: Neither blocks nor log segments found. Creating new log.
I20260810 22:37:23.509334 3382 log.cc:824] T 9a2cb51d98444688a200c5d0278fcbb5 P 32bc0a72e0f7471186c4b04c2e8a0422: Log is configured to *not* fsync() on all Append() calls
I20260810 22:37:23.509331 3384 log.cc:824] T 9a2cb51d98444688a200c5d0278fcbb5 P f1e19b4ab0404fbeb1208e06ed4bf0ee: Log is configured to *not* fsync() on all Append() calls
I20260810 22:37:23.511404 3384 tablet_bootstrap.cc:492] T 9a2cb51d98444688a200c5d0278fcbb5 P f1e19b4ab0404fbeb1208e06ed4bf0ee: No bootstrap required, opened a new log
I20260810 22:37:23.511636 3384 ts_tablet_manager.cc:1397] T 9a2cb51d98444688a200c5d0278fcbb5 P f1e19b4ab0404fbeb1208e06ed4bf0ee: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 22:37:23.512892 3382 tablet_bootstrap.cc:492] T 9a2cb51d98444688a200c5d0278fcbb5 P 32bc0a72e0f7471186c4b04c2e8a0422: No bootstrap required, opened a new log
I20260810 22:37:23.513090 3382 ts_tablet_manager.cc:1397] T 9a2cb51d98444688a200c5d0278fcbb5 P 32bc0a72e0f7471186c4b04c2e8a0422: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 22:37:23.515676 3384 raft_consensus.cc:348] T 9a2cb51d98444688a200c5d0278fcbb5 P f1e19b4ab0404fbeb1208e06ed4bf0ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e19b4ab0404fbeb1208e06ed4bf0ee" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 38143 } } peers { permanent_uuid: "32bc0a72e0f7471186c4b04c2e8a0422" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 34433 } } peers { permanent_uuid: "f7df0146ce9c40c9a2d24d9834a2125f" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 36229 } }
I20260810 22:37:23.515996 3384 raft_consensus.cc:374] T 9a2cb51d98444688a200c5d0278fcbb5 P f1e19b4ab0404fbeb1208e06ed4bf0ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:37:23.516055 3384 raft_consensus.cc:729] T 9a2cb51d98444688a200c5d0278fcbb5 P f1e19b4ab0404fbeb1208e06ed4bf0ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e19b4ab0404fbeb1208e06ed4bf0ee, State: Initialized, Role: FOLLOWER
W20260810 22:37:23.516088 3062 tablet.cc:2369] T 9a2cb51d98444688a200c5d0278fcbb5 P f7df0146ce9c40c9a2d24d9834a2125f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:37:23.516531 3384 consensus_queue.cc:260] T 9a2cb51d98444688a200c5d0278fcbb5 P f1e19b4ab0404fbeb1208e06ed4bf0ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e19b4ab0404fbeb1208e06ed4bf0ee" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 38143 } } peers { permanent_uuid: "32bc0a72e0f7471186c4b04c2e8a0422" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 34433 } } peers { permanent_uuid: "f7df0146ce9c40c9a2d24d9834a2125f" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 36229 } }
I20260810 22:37:23.516963 3382 raft_consensus.cc:348] T 9a2cb51d98444688a200c5d0278fcbb5 P 32bc0a72e0f7471186c4b04c2e8a0422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e19b4ab0404fbeb1208e06ed4bf0ee" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 38143 } } peers { permanent_uuid: "32bc0a72e0f7471186c4b04c2e8a0422" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 34433 } } peers { permanent_uuid: "f7df0146ce9c40c9a2d24d9834a2125f" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 36229 } }
I20260810 22:37:23.517225 3382 raft_consensus.cc:374] T 9a2cb51d98444688a200c5d0278fcbb5 P 32bc0a72e0f7471186c4b04c2e8a0422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:37:23.517293 3382 raft_consensus.cc:729] T 9a2cb51d98444688a200c5d0278fcbb5 P 32bc0a72e0f7471186c4b04c2e8a0422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32bc0a72e0f7471186c4b04c2e8a0422, State: Initialized, Role: FOLLOWER
I20260810 22:37:23.517309 3366 heartbeater.cc:503] Master 127.16.233.126:39115 was elected leader, sending a full tablet report...
I20260810 22:37:23.517704 3384 ts_tablet_manager.cc:1428] T 9a2cb51d98444688a200c5d0278fcbb5 P f1e19b4ab0404fbeb1208e06ed4bf0ee: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260810 22:37:23.517709 3382 consensus_queue.cc:260] T 9a2cb51d98444688a200c5d0278fcbb5 P 32bc0a72e0f7471186c4b04c2e8a0422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e19b4ab0404fbeb1208e06ed4bf0ee" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 38143 } } peers { permanent_uuid: "32bc0a72e0f7471186c4b04c2e8a0422" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 34433 } } peers { permanent_uuid: "f7df0146ce9c40c9a2d24d9834a2125f" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 36229 } }
I20260810 22:37:23.518551 3218 heartbeater.cc:503] Master 127.16.233.126:39115 was elected leader, sending a full tablet report...
I20260810 22:37:23.518662 3382 ts_tablet_manager.cc:1428] T 9a2cb51d98444688a200c5d0278fcbb5 P 32bc0a72e0f7471186c4b04c2e8a0422: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260810 22:37:23.666409 3389 raft_consensus.cc:482] T 9a2cb51d98444688a200c5d0278fcbb5 P 32bc0a72e0f7471186c4b04c2e8a0422 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:37:23.666687 3389 raft_consensus.cc:504] T 9a2cb51d98444688a200c5d0278fcbb5 P 32bc0a72e0f7471186c4b04c2e8a0422 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e19b4ab0404fbeb1208e06ed4bf0ee" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 38143 } } peers { permanent_uuid: "32bc0a72e0f7471186c4b04c2e8a0422" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 34433 } } peers { permanent_uuid: "f7df0146ce9c40c9a2d24d9834a2125f" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 36229 } }
I20260810 22:37:23.667979 3389 leader_election.cc:288] T 9a2cb51d98444688a200c5d0278fcbb5 P 32bc0a72e0f7471186c4b04c2e8a0422 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e19b4ab0404fbeb1208e06ed4bf0ee (127.16.233.67:38143), f7df0146ce9c40c9a2d24d9834a2125f (127.16.233.65:36229)
I20260810 22:37:23.672766 3220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:23.668420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.67:38143 (local address 127.16.233.66:36761)
0810 22:37:23.668562 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0810 22:37:23.668581 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 22:37:23.668688 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:37:23.669128 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:37:23.669131 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:37:23.669156 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:37:23.669297 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:37:23.669306 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:37:23.670306 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:37:23.670309 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:37:23.671499 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:37:23.671510 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:37:23.672357 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:37:23.672360 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:37:23.672457 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:37:23.672479 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 22:37:23.672551 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 22:37:23.672611 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260810 22:37:23.672855 3377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:23.668693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:36761 (local address 127.16.233.67:38143)
0810 22:37:23.668859 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 22:37:23.668864 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:37:23.668883 (+ 19us) server_negotiation.cc:407] Connection header received
0810 22:37:23.668950 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:37:23.668953 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:37:23.669014 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:37:23.669108 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:37:23.669426 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:37:23.670143 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:37:23.671661 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:37:23.672222 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:37:23.672360 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:37:23.672417 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 22:37:23.672603 (+ 186us) server_negotiation.cc:299] Negotiation successful
0810 22:37:23.672698 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 22:37:23.673373 3391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:23.668717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.65:36229 (local address 127.16.233.66:38555)
0810 22:37:23.669133 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0810 22:37:23.669148 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:37:23.669226 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:37:23.669682 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:37:23.669685 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:37:23.669705 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:37:23.669821 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:37:23.669827 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:37:23.670681 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:37:23.670684 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:37:23.671817 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:37:23.671824 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:37:23.673025 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:37:23.673029 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:37:23.673147 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:37:23.673165 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 22:37:23.673222 (+ 57us) client_negotiation.cc:240] Negotiation successful
0810 22:37:23.673275 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":200,"threads_started":1}
I20260810 22:37:23.673527 3392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:23.668903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:38555 (local address 127.16.233.65:36229)
0810 22:37:23.669181 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0810 22:37:23.669186 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:37:23.669203 (+ 17us) server_negotiation.cc:407] Connection header received
0810 22:37:23.669345 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:37:23.669368 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:37:23.669446 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:37:23.669574 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:37:23.669961 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:37:23.670557 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:37:23.671944 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:37:23.672629 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:37:23.672751 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:37:23.673033 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0810 22:37:23.673274 (+ 241us) server_negotiation.cc:299] Negotiation successful
0810 22:37:23.673401 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":81,"threads_started":1}
I20260810 22:37:23.673645 3297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a2cb51d98444688a200c5d0278fcbb5" candidate_uuid: "32bc0a72e0f7471186c4b04c2e8a0422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e19b4ab0404fbeb1208e06ed4bf0ee" is_pre_election: true
I20260810 22:37:23.674031 3297 raft_consensus.cc:2454] T 9a2cb51d98444688a200c5d0278fcbb5 P f1e19b4ab0404fbeb1208e06ed4bf0ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32bc0a72e0f7471186c4b04c2e8a0422 in term 0.
I20260810 22:37:23.674155 3012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a2cb51d98444688a200c5d0278fcbb5" candidate_uuid: "32bc0a72e0f7471186c4b04c2e8a0422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7df0146ce9c40c9a2d24d9834a2125f" is_pre_election: true
I20260810 22:37:23.674465 3012 raft_consensus.cc:2454] T 9a2cb51d98444688a200c5d0278fcbb5 P f7df0146ce9c40c9a2d24d9834a2125f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32bc0a72e0f7471186c4b04c2e8a0422 in term 0.
I20260810 22:37:23.674599 3094 leader_election.cc:302] T 9a2cb51d98444688a200c5d0278fcbb5 P 32bc0a72e0f7471186c4b04c2e8a0422 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32bc0a72e0f7471186c4b04c2e8a0422, f1e19b4ab0404fbeb1208e06ed4bf0ee; no voters:
I20260810 22:37:23.674950 3389 raft_consensus.cc:2781] T 9a2cb51d98444688a200c5d0278fcbb5 P 32bc0a72e0f7471186c4b04c2e8a0422 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:37:23.675058 3389 raft_consensus.cc:482] T 9a2cb51d98444688a200c5d0278fcbb5 P 32bc0a72e0f7471186c4b04c2e8a0422 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:37:23.675122 3389 raft_consensus.cc:3037] T 9a2cb51d98444688a200c5d0278fcbb5 P 32bc0a72e0f7471186c4b04c2e8a0422 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:37:23.676240 3389 raft_consensus.cc:504] T 9a2cb51d98444688a200c5d0278fcbb5 P 32bc0a72e0f7471186c4b04c2e8a0422 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e19b4ab0404fbeb1208e06ed4bf0ee" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 38143 } } peers { permanent_uuid: "32bc0a72e0f7471186c4b04c2e8a0422" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 34433 } } peers { permanent_uuid: "f7df0146ce9c40c9a2d24d9834a2125f" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 36229 } }
I20260810 22:37:23.676621 3389 leader_election.cc:288] T 9a2cb51d98444688a200c5d0278fcbb5 P 32bc0a72e0f7471186c4b04c2e8a0422 [CANDIDATE]: Term 1 election: Requested vote from peers f1e19b4ab0404fbeb1208e06ed4bf0ee (127.16.233.67:38143), f7df0146ce9c40c9a2d24d9834a2125f (127.16.233.65:36229)
I20260810 22:37:23.677203 3297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a2cb51d98444688a200c5d0278fcbb5" candidate_uuid: "32bc0a72e0f7471186c4b04c2e8a0422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e19b4ab0404fbeb1208e06ed4bf0ee"
I20260810 22:37:23.677241 3012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a2cb51d98444688a200c5d0278fcbb5" candidate_uuid: "32bc0a72e0f7471186c4b04c2e8a0422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7df0146ce9c40c9a2d24d9834a2125f"
I20260810 22:37:23.677377 3297 raft_consensus.cc:3037] T 9a2cb51d98444688a200c5d0278fcbb5 P f1e19b4ab0404fbeb1208e06ed4bf0ee [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:37:23.677397 3012 raft_consensus.cc:3037] T 9a2cb51d98444688a200c5d0278fcbb5 P f7df0146ce9c40c9a2d24d9834a2125f [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:37:23.678519 3012 raft_consensus.cc:2454] T 9a2cb51d98444688a200c5d0278fcbb5 P f7df0146ce9c40c9a2d24d9834a2125f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32bc0a72e0f7471186c4b04c2e8a0422 in term 1.
I20260810 22:37:23.678805 3297 raft_consensus.cc:2454] T 9a2cb51d98444688a200c5d0278fcbb5 P f1e19b4ab0404fbeb1208e06ed4bf0ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32bc0a72e0f7471186c4b04c2e8a0422 in term 1.
I20260810 22:37:23.678900 3094 leader_election.cc:302] T 9a2cb51d98444688a200c5d0278fcbb5 P 32bc0a72e0f7471186c4b04c2e8a0422 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32bc0a72e0f7471186c4b04c2e8a0422, f7df0146ce9c40c9a2d24d9834a2125f; no voters:
I20260810 22:37:23.679145 3389 raft_consensus.cc:2781] T 9a2cb51d98444688a200c5d0278fcbb5 P 32bc0a72e0f7471186c4b04c2e8a0422 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:37:23.679389 3389 raft_consensus.cc:686] T 9a2cb51d98444688a200c5d0278fcbb5 P 32bc0a72e0f7471186c4b04c2e8a0422 [term 1 LEADER]: Becoming Leader. State: Replica: 32bc0a72e0f7471186c4b04c2e8a0422, State: Running, Role: LEADER
I20260810 22:37:23.679761 3389 consensus_queue.cc:237] T 9a2cb51d98444688a200c5d0278fcbb5 P 32bc0a72e0f7471186c4b04c2e8a0422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e19b4ab0404fbeb1208e06ed4bf0ee" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 38143 } } peers { permanent_uuid: "32bc0a72e0f7471186c4b04c2e8a0422" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 34433 } } peers { permanent_uuid: "f7df0146ce9c40c9a2d24d9834a2125f" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 36229 } }
I20260810 22:37:23.683425 2858 catalog_manager.cc:5495] T 9a2cb51d98444688a200c5d0278fcbb5 P 32bc0a72e0f7471186c4b04c2e8a0422 reported cstate change: term changed from 0 to 1, leader changed from <none> to 32bc0a72e0f7471186c4b04c2e8a0422 (127.16.233.66). New cstate: current_term: 1 leader_uuid: "32bc0a72e0f7471186c4b04c2e8a0422" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e19b4ab0404fbeb1208e06ed4bf0ee" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 38143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32bc0a72e0f7471186c4b04c2e8a0422" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 34433 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7df0146ce9c40c9a2d24d9834a2125f" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 36229 } health_report { overall_health: UNKNOWN } } }
W20260810 22:37:23.690519 3367 tablet.cc:2369] T 9a2cb51d98444688a200c5d0278fcbb5 P f1e19b4ab0404fbeb1208e06ed4bf0ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:37:23.696903 3379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:23.693104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48400 (local address 127.16.233.66:34433)
0810 22:37:23.693257 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 22:37:23.693262 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:37:23.693279 (+ 17us) server_negotiation.cc:407] Connection header received
0810 22:37:23.693402 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:37:23.693408 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:37:23.693495 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:37:23.693619 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:37:23.693877 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:37:23.694402 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:37:23.695268 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:37:23.695499 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:37:23.695614 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:37:23.695959 (+ 345us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:37:23.696420 (+ 461us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:37:23.696582 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0810 22:37:23.696629 (+ 47us) server_negotiation.cc:299] Negotiation successful
0810 22:37:23.696696 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 22:37:23.703981 3012 raft_consensus.cc:1261] T 9a2cb51d98444688a200c5d0278fcbb5 P f7df0146ce9c40c9a2d24d9834a2125f [term 1 FOLLOWER]: Refusing update from remote peer 32bc0a72e0f7471186c4b04c2e8a0422: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:37:23.704020 3297 raft_consensus.cc:1261] T 9a2cb51d98444688a200c5d0278fcbb5 P f1e19b4ab0404fbeb1208e06ed4bf0ee [term 1 FOLLOWER]: Refusing update from remote peer 32bc0a72e0f7471186c4b04c2e8a0422: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:37:23.704957 3389 consensus_queue.cc:1035] T 9a2cb51d98444688a200c5d0278fcbb5 P 32bc0a72e0f7471186c4b04c2e8a0422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e19b4ab0404fbeb1208e06ed4bf0ee" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 38143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:37:23.705422 3393 consensus_queue.cc:1035] T 9a2cb51d98444688a200c5d0278fcbb5 P 32bc0a72e0f7471186c4b04c2e8a0422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7df0146ce9c40c9a2d24d9834a2125f" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 36229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:37:23.713987 3400 mvcc.cc:205] Tried to move back new op lower bound from 7317100313401618432 to 7317100313315721216. Current Snapshot: MvccSnapshot[applied={T|T < 7317100313401618432}]
I20260810 22:37:23.714214 3399 mvcc.cc:205] Tried to move back new op lower bound from 7317100313401618432 to 7317100313315721216. Current Snapshot: MvccSnapshot[applied={T|T < 7317100313401618432}]
I20260810 22:37:23.722358 2858 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:37:23.722580 2858 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:37:23.725307 2858 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c7109630f3354b35aa075a1f467262d4: Sending DeleteTablet for 3 replicas of tablet 9a2cb51d98444688a200c5d0278fcbb5
I20260810 22:37:23.726249 3146 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a2cb51d98444688a200c5d0278fcbb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:37:23 UTC) from {username='kudu', principal='kudu/127.16.233.126@KRBTEST.COM'} at 127.0.0.1:48394
I20260810 22:37:23.726300 2991 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a2cb51d98444688a200c5d0278fcbb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:37:23 UTC) from {username='kudu', principal='kudu/127.16.233.126@KRBTEST.COM'} at 127.0.0.1:39098
I20260810 22:37:23.726243 3282 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a2cb51d98444688a200c5d0278fcbb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:37:23 UTC) from {username='kudu', principal='kudu/127.16.233.126@KRBTEST.COM'} at 127.0.0.1:45806
I20260810 22:37:23.726828 3409 tablet_replica.cc:330] T 9a2cb51d98444688a200c5d0278fcbb5 P f1e19b4ab0404fbeb1208e06ed4bf0ee: stopping tablet replica
I20260810 22:37:23.727013 3408 tablet_replica.cc:330] T 9a2cb51d98444688a200c5d0278fcbb5 P f7df0146ce9c40c9a2d24d9834a2125f: stopping tablet replica
I20260810 22:37:23.727183 3408 raft_consensus.cc:2229] T 9a2cb51d98444688a200c5d0278fcbb5 P f7df0146ce9c40c9a2d24d9834a2125f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:37:23.727218 3409 raft_consensus.cc:2229] T 9a2cb51d98444688a200c5d0278fcbb5 P f1e19b4ab0404fbeb1208e06ed4bf0ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:37:23.727226 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 2900
I20260810 22:37:23.727456 3408 raft_consensus.cc:2258] T 9a2cb51d98444688a200c5d0278fcbb5 P f7df0146ce9c40c9a2d24d9834a2125f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:37:23.727614 3409 raft_consensus.cc:2258] T 9a2cb51d98444688a200c5d0278fcbb5 P f1e19b4ab0404fbeb1208e06ed4bf0ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:37:23.728724 3410 tablet_replica.cc:330] T 9a2cb51d98444688a200c5d0278fcbb5 P 32bc0a72e0f7471186c4b04c2e8a0422: stopping tablet replica
I20260810 22:37:23.729076 3410 raft_consensus.cc:2229] T 9a2cb51d98444688a200c5d0278fcbb5 P 32bc0a72e0f7471186c4b04c2e8a0422 [term 1 LEADER]: Raft consensus shutting down.
I20260810 22:37:23.729583 3410 raft_consensus.cc:2258] T 9a2cb51d98444688a200c5d0278fcbb5 P 32bc0a72e0f7471186c4b04c2e8a0422 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:37:23.731232 3409 ts_tablet_manager.cc:1905] T 9a2cb51d98444688a200c5d0278fcbb5 P f1e19b4ab0404fbeb1208e06ed4bf0ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:37:23.731231 3410 ts_tablet_manager.cc:1905] T 9a2cb51d98444688a200c5d0278fcbb5 P 32bc0a72e0f7471186c4b04c2e8a0422: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:37:23.734184 3410 ts_tablet_manager.cc:1918] T 9a2cb51d98444688a200c5d0278fcbb5 P 32bc0a72e0f7471186c4b04c2e8a0422: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:37:23.734182 3409 ts_tablet_manager.cc:1918] T 9a2cb51d98444688a200c5d0278fcbb5 P f1e19b4ab0404fbeb1208e06ed4bf0ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:37:23.734299 3410 log.cc:1196] T 9a2cb51d98444688a200c5d0278fcbb5 P 32bc0a72e0f7471186c4b04c2e8a0422: Deleting WAL directory at /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-1/wal/wals/9a2cb51d98444688a200c5d0278fcbb5
I20260810 22:37:23.734483 3409 log.cc:1196] T 9a2cb51d98444688a200c5d0278fcbb5 P f1e19b4ab0404fbeb1208e06ed4bf0ee: Deleting WAL directory at /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-2/wal/wals/9a2cb51d98444688a200c5d0278fcbb5
I20260810 22:37:23.734663 3410 ts_tablet_manager.cc:1939] T 9a2cb51d98444688a200c5d0278fcbb5 P 32bc0a72e0f7471186c4b04c2e8a0422: Deleting consensus metadata
I20260810 22:37:23.735448 3409 ts_tablet_manager.cc:1939] T 9a2cb51d98444688a200c5d0278fcbb5 P f1e19b4ab0404fbeb1208e06ed4bf0ee: Deleting consensus metadata
I20260810 22:37:23.737172 2844 catalog_manager.cc:4841] TS f1e19b4ab0404fbeb1208e06ed4bf0ee (127.16.233.67:38143): tablet 9a2cb51d98444688a200c5d0278fcbb5 (table test-table [id=569bb4ae1f50461e8b0f557e424dfda5]) successfully deleted
I20260810 22:37:23.739178 2842 catalog_manager.cc:4841] TS 32bc0a72e0f7471186c4b04c2e8a0422 (127.16.233.66:34433): tablet 9a2cb51d98444688a200c5d0278fcbb5 (table test-table [id=569bb4ae1f50461e8b0f557e424dfda5]) successfully deleted
W20260810 22:37:23.741276 2844 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.233.65:36229 (error 108)
I20260810 22:37:23.741647 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 3067
I20260810 22:37:23.743086 3071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:23.742583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.65:36229 (local address 127.0.0.1:39100)
0810 22:37:23.742889 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0810 22:37:23.742974 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.65:36229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185}
W20260810 22:37:23.743290 2844 catalog_manager.cc:4584] TS f7df0146ce9c40c9a2d24d9834a2125f (127.16.233.65:36229): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9a2cb51d98444688a200c5d0278fcbb5: Network error: Client connection negotiation failed: client connection to 127.16.233.65:36229: connect: Connection refused (error 111)
I20260810 22:37:23.751714 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 3221
I20260810 22:37:23.762478 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 2823
I20260810 22:37:43.773626 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:39115
--webserver_interface=127.16.233.126
--webserver_port=43295
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab
--principal=kudu/127.16.233.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:38849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.126:39115
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:37:43.893669 3413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:37:43.894064 3413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:37:43.894168 3413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:37:43.898458 3413 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 22:37:43.898566 3413 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 22:37:43.898592 3413 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 22:37:43.898617 3413 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:37:43.898665 3413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:37:43.898707 3413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:37:43.898746 3413 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:37:43.898782 3413 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:37:43.904467 3413 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:38849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.16.233.126:39115
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab
--principal=kudu/127.16.233.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:39115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.126
--webserver_port=43295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:37:43.905757 3413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:37:43.906898 3413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 22:37:43.917073 3413 server_base.cc:1034] running on GCE node
W20260810 22:37:43.917145 3418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:37:43.917094 3419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:37:43.917052 3421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:37:43.917944 3413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:37:43.919229 3413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:37:43.920488 3413 hybrid_clock.cc:648] HybridClock initialized: now 1786401463920471 us; error 33 us; skew 500 ppm
Aug 10 22:37:43 dist-test-slave-77v9 krb5kdc[2204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401463, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:37:43.929925 3413 init.cc:377] Logged in from keytab as kudu/127.16.233.126@KRBTEST.COM (short username kudu)
I20260810 22:37:43.931087 3413 webserver.cc:460] Webserver started at http://127.16.233.126:43295/ using document root <none> and password file <none>
I20260810 22:37:43.931774 3413 fs_manager.cc:359] Metadata directory not provided
I20260810 22:37:43.931890 3413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:37:43.936395 3413 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260810 22:37:43.939069 3430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:37:43.940578 3413 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 22:37:43.940754 3413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/master-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/master-0/wal
uuid: "c7109630f3354b35aa075a1f467262d4"
format_stamp: "Formatted at 2026-08-10 22:37:01 on dist-test-slave-77v9"
server_key: "2eff5433112c9d11164a106669587b59"
server_key_iv: "8d8791d9ad2768b560b36d7faa4cbe00"
server_key_version: "encryptionkey@0"
I20260810 22:37:43.941483 3413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:37:43.965416 3413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:37:43.969295 3413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:37:43.969573 3413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:37:43.978185 3413 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.126:39115
I20260810 22:37:43.978529 3489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.126:39115 every 8 connection(s)
I20260810 22:37:43.979519 3413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
I20260810 22:37:43.982939 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 3413
I20260810 22:37:43.983511 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.65:36229
--local_ip_for_outbound_sockets=127.16.233.65
--tserver_master_addrs=127.16.233.126:39115
--webserver_port=38839
--webserver_interface=127.16.233.65
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab
--principal=kudu/127.16.233.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:38849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 22:37:43.985045 3490 sys_catalog.cc:263] Verifying existing consensus state
I20260810 22:37:43.987264 3490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7109630f3354b35aa075a1f467262d4: Bootstrap starting.
I20260810 22:37:44.008389 3490 log.cc:824] T 00000000000000000000000000000000 P c7109630f3354b35aa075a1f467262d4: Log is configured to *not* fsync() on all Append() calls
I20260810 22:37:44.021451 3490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7109630f3354b35aa075a1f467262d4: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 22:37:44.021963 3490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7109630f3354b35aa075a1f467262d4: Bootstrap complete.
I20260810 22:37:44.026818 3490 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7109630f3354b35aa075a1f467262d4 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7109630f3354b35aa075a1f467262d4" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39115 } }
I20260810 22:37:44.027271 3490 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7109630f3354b35aa075a1f467262d4 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7109630f3354b35aa075a1f467262d4, State: Initialized, Role: FOLLOWER
I20260810 22:37:44.027814 3490 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7109630f3354b35aa075a1f467262d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7109630f3354b35aa075a1f467262d4" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39115 } }
I20260810 22:37:44.027949 3490 raft_consensus.cc:388] T 00000000000000000000000000000000 P c7109630f3354b35aa075a1f467262d4 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:37:44.028064 3490 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7109630f3354b35aa075a1f467262d4 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:37:44.028192 3490 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7109630f3354b35aa075a1f467262d4 [term 2 FOLLOWER]: Advancing to term 3
I20260810 22:37:44.030561 3490 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7109630f3354b35aa075a1f467262d4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7109630f3354b35aa075a1f467262d4" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39115 } }
I20260810 22:37:44.030943 3490 leader_election.cc:302] T 00000000000000000000000000000000 P c7109630f3354b35aa075a1f467262d4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c7109630f3354b35aa075a1f467262d4; no voters:
I20260810 22:37:44.031363 3490 leader_election.cc:288] T 00000000000000000000000000000000 P c7109630f3354b35aa075a1f467262d4 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260810 22:37:44.031471 3494 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7109630f3354b35aa075a1f467262d4 [term 3 FOLLOWER]: Leader election won for term 3
I20260810 22:37:44.031733 3494 raft_consensus.cc:686] T 00000000000000000000000000000000 P c7109630f3354b35aa075a1f467262d4 [term 3 LEADER]: Becoming Leader. State: Replica: c7109630f3354b35aa075a1f467262d4, State: Running, Role: LEADER
I20260810 22:37:44.032145 3494 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7109630f3354b35aa075a1f467262d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7109630f3354b35aa075a1f467262d4" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39115 } }
I20260810 22:37:44.032929 3490 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7109630f3354b35aa075a1f467262d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:37:44.033283 3494 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7109630f3354b35aa075a1f467262d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "c7109630f3354b35aa075a1f467262d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7109630f3354b35aa075a1f467262d4" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39115 } } }
I20260810 22:37:44.033622 3494 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7109630f3354b35aa075a1f467262d4 [sys.catalog]: This master's current role is: LEADER
I20260810 22:37:44.033715 3495 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7109630f3354b35aa075a1f467262d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7109630f3354b35aa075a1f467262d4. Latest consensus state: current_term: 3 leader_uuid: "c7109630f3354b35aa075a1f467262d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7109630f3354b35aa075a1f467262d4" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39115 } } }
I20260810 22:37:44.033977 3495 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7109630f3354b35aa075a1f467262d4 [sys.catalog]: This master's current role is: LEADER
I20260810 22:37:44.034168 3500 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:37:44.039994 3500 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:37:44.041124 3500 catalog_manager.cc:1259] Loaded cluster ID: b3ab420c4a7442a2bc0fcda8f9d00e2b
I20260810 22:37:44.041198 3500 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:37:44.044062 3500 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:37:44.045024 3500 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c7109630f3354b35aa075a1f467262d4: Loaded TSK: 0
I20260810 22:37:44.045126 3500 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c7109630f3354b35aa075a1f467262d4: Loaded TSK: 1
F20260810 22:37:44.045995 3500 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f78d2a3d9ad google::LogMessage::Fail() at ??:0
@ 0x7f78d2a4088a google::LogMessage::SendToLog() at ??:0
@ 0x7f78d2a3d4de google::LogMessage::Flush() at ??:0
@ 0x7f78d2a3ea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f78d570118d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f78d56f9f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f78dd3a27bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f78dd37c99c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f78dd37d178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f78dd3bedf8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f78dd3dd82b std::function<>::operator()() at ??:0
@ 0x7f78dd37cb4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f78dd37df3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f78dd37a579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f78dd3be640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f78d4c8f3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f78d4c8fc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f78d4c913f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f78d4c80512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f78d62c46db start_thread at ??:0
@ 0x7f78d153b61f clone at ??:0
W20260810 22:37:44.146864 3492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:37:44.147300 3492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:37:44.147378 3492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:37:44.154966 3492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:37:44.155052 3492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:37:44.155166 3492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.65
I20260810 22:37:44.165314 3492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:38849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab
--principal=kudu/127.16.233.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.65:36229
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.233.65
--webserver_port=38839
--tserver_master_addrs=127.16.233.126:39115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3492
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.65
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:37:44.166880 3492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:37:44.168354 3492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:37:44.182684 3547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:37:44.182768 3545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:37:44.182909 3492 server_base.cc:1034] running on GCE node
W20260810 22:37:44.183029 3544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:37:44.183529 3492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:37:44.184453 3492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:37:44.185664 3492 hybrid_clock.cc:648] HybridClock initialized: now 1786401464185643 us; error 42 us; skew 500 ppm
Aug 10 22:37:44 dist-test-slave-77v9 krb5kdc[2204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401464, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:37:44.188614 3492 init.cc:377] Logged in from keytab as kudu/127.16.233.65@KRBTEST.COM (short username kudu)
I20260810 22:37:44.189600 3492 webserver.cc:460] Webserver started at http://127.16.233.65:38839/ using document root <none> and password file <none>
I20260810 22:37:44.190186 3492 fs_manager.cc:359] Metadata directory not provided
I20260810 22:37:44.190249 3492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:37:44.194553 3492 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260810 22:37:44.197448 3554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:37:44.199246 3492 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:37:44.199409 3492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-0/wal
uuid: "f7df0146ce9c40c9a2d24d9834a2125f"
format_stamp: "Formatted at 2026-08-10 22:37:01 on dist-test-slave-77v9"
server_key: "b284566687ddcb1a6eca3bfc282fa8db"
server_key_iv: "f6cb8a8b6cbd9908e77830f03c31c8c3"
server_key_version: "encryptionkey@0"
I20260810 22:37:44.200026 3492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:37:44.212893 3492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:37:44.216487 3492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:37:44.216718 3492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:37:44.217453 3492 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:37:44.218720 3561 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 22:37:44.223174 3492 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 22:37:44.223261 3492 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260810 22:37:44.223359 3492 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 22:37:44.225354 3492 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 22:37:44.225435 3492 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260810 22:37:44.226109 3561 tablet_bootstrap.cc:492] T 9a2cb51d98444688a200c5d0278fcbb5 P f7df0146ce9c40c9a2d24d9834a2125f: Bootstrap starting.
I20260810 22:37:44.237057 3492 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.65:36229
I20260810 22:37:44.238117 3492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
I20260810 22:37:44.239275 3689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.65:36229 every 8 connection(s)
I20260810 22:37:44.243942 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 3492
I20260810 22:37:44.244498 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.66:34433
--local_ip_for_outbound_sockets=127.16.233.66
--tserver_master_addrs=127.16.233.126:39115
--webserver_port=41565
--webserver_interface=127.16.233.66
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab
--principal=kudu/127.16.233.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:38849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 22:37:44.251909 3561 log.cc:824] T 9a2cb51d98444688a200c5d0278fcbb5 P f7df0146ce9c40c9a2d24d9834a2125f: Log is configured to *not* fsync() on all Append() calls
I20260810 22:37:44.262625 3561 tablet_bootstrap.cc:492] T 9a2cb51d98444688a200c5d0278fcbb5 P f7df0146ce9c40c9a2d24d9834a2125f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 22:37:44.263166 3561 tablet_bootstrap.cc:492] T 9a2cb51d98444688a200c5d0278fcbb5 P f7df0146ce9c40c9a2d24d9834a2125f: Bootstrap complete.
I20260810 22:37:44.265033 3561 ts_tablet_manager.cc:1397] T 9a2cb51d98444688a200c5d0278fcbb5 P f7df0146ce9c40c9a2d24d9834a2125f: Time spent bootstrapping tablet: real 0.039s user 0.021s sys 0.017s
I20260810 22:37:44.269116 3561 raft_consensus.cc:348] T 9a2cb51d98444688a200c5d0278fcbb5 P f7df0146ce9c40c9a2d24d9834a2125f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e19b4ab0404fbeb1208e06ed4bf0ee" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 38143 } } peers { permanent_uuid: "32bc0a72e0f7471186c4b04c2e8a0422" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 34433 } } peers { permanent_uuid: "f7df0146ce9c40c9a2d24d9834a2125f" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 36229 } }
I20260810 22:37:44.269742 3561 raft_consensus.cc:729] T 9a2cb51d98444688a200c5d0278fcbb5 P f7df0146ce9c40c9a2d24d9834a2125f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7df0146ce9c40c9a2d24d9834a2125f, State: Initialized, Role: FOLLOWER
I20260810 22:37:44.270299 3561 consensus_queue.cc:260] T 9a2cb51d98444688a200c5d0278fcbb5 P f7df0146ce9c40c9a2d24d9834a2125f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e19b4ab0404fbeb1208e06ed4bf0ee" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 38143 } } peers { permanent_uuid: "32bc0a72e0f7471186c4b04c2e8a0422" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 34433 } } peers { permanent_uuid: "f7df0146ce9c40c9a2d24d9834a2125f" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 36229 } }
I20260810 22:37:44.271476 3561 ts_tablet_manager.cc:1428] T 9a2cb51d98444688a200c5d0278fcbb5 P f7df0146ce9c40c9a2d24d9834a2125f: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
W20260810 22:37:44.351014 3692 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:37:44.240455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39115 (local address 127.16.233.65:38939)
0810 22:37:44.241043 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0810 22:37:44.241101 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 22:37:44.241982 (+ 881us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:37:44.349930 (+107948us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39115: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":168,"threads_started":1}
W20260810 22:37:44.352063 3690 heartbeater.cc:644] Failed to heartbeat to 127.16.233.126:39115 (0 consecutive failures): Network error: Failed to ping master at 127.16.233.126:39115: Client connection negotiation failed: client connection to 127.16.233.126:39115: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 22:37:44.353073 3692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:44.352609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39115 (local address 127.16.233.65:44985)
0810 22:37:44.352860 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0810 22:37:44.352969 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260810 22:37:44.353860 3692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:44.353566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39115 (local address 127.16.233.65:52107)
0810 22:37:44.353699 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0810 22:37:44.353766 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260810 22:37:44.354069 3690 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 22:37:44.382330 3694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:37:44.382606 3694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:37:44.382649 3694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:37:44.386775 3694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:37:44.386837 3694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:37:44.386896 3694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.66
I20260810 22:37:44.392191 3694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:38849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab
--principal=kudu/127.16.233.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.66:34433
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.233.66
--webserver_port=41565
--tserver_master_addrs=127.16.233.126:39115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.66
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:37:44.393469 3694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:37:44.394515 3694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:37:44.404558 3702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:37:44.404593 3706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:37:44.404745 3694 server_base.cc:1034] running on GCE node
W20260810 22:37:44.404567 3704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:37:44.405365 3694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:37:44.406036 3694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:37:44.407202 3694 hybrid_clock.cc:648] HybridClock initialized: now 1786401464407177 us; error 38 us; skew 500 ppm
Aug 10 22:37:44 dist-test-slave-77v9 krb5kdc[2204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401464, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:37:44.410226 3694 init.cc:377] Logged in from keytab as kudu/127.16.233.66@KRBTEST.COM (short username kudu)
I20260810 22:37:44.411185 3694 webserver.cc:460] Webserver started at http://127.16.233.66:41565/ using document root <none> and password file <none>
I20260810 22:37:44.411769 3694 fs_manager.cc:359] Metadata directory not provided
I20260810 22:37:44.411834 3694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:37:44.415932 3694 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 22:37:44.418249 3714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:37:44.419529 3694 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 22:37:44.419634 3694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-1/wal
uuid: "32bc0a72e0f7471186c4b04c2e8a0422"
format_stamp: "Formatted at 2026-08-10 22:37:02 on dist-test-slave-77v9"
server_key: "90aac31c1a5f558d2a48a0697537b93a"
server_key_iv: "f08935cd48ca5afccf11d200362e9c96"
server_key_version: "encryptionkey@0"
I20260810 22:37:44.420245 3694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:37:44.436940 3694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:37:44.440285 3694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:37:44.440696 3694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:37:44.441563 3694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:37:44.442536 3694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:37:44.442600 3694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:37:44.442674 3694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:37:44.442718 3694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:37:44.454397 3694 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.66:34433
I20260810 22:37:44.454473 3846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.66:34433 every 8 connection(s)
I20260810 22:37:44.455703 3694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
I20260810 22:37:44.460211 3850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:44.458309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39115 (local address 127.16.233.66:53611)
0810 22:37:44.458900 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0810 22:37:44.459022 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":136,"threads_started":1}
W20260810 22:37:44.460700 3847 heartbeater.cc:644] Failed to heartbeat to 127.16.233.126:39115 (0 consecutive failures): Network error: Failed to ping master at 127.16.233.126:39115: Client connection negotiation failed: client connection to 127.16.233.126:39115: connect: Connection refused (error 111)
I20260810 22:37:44.461385 3850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:44.461163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39115 (local address 127.16.233.66:59161)
0810 22:37:44.461276 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0810 22:37:44.461328 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260810 22:37:44.462052 3850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:44.461822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39115 (local address 127.16.233.66:39879)
0810 22:37:44.461953 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0810 22:37:44.462002 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260810 22:37:44.462277 3847 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 22:37:44.464946 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 3694
I20260810 22:37:44.465401 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.67:38143
--local_ip_for_outbound_sockets=127.16.233.67
--tserver_master_addrs=127.16.233.126:39115
--webserver_port=35907
--webserver_interface=127.16.233.67
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab
--principal=kudu/127.16.233.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:38849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:37:44.583441 3851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:37:44.583753 3851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:37:44.583801 3851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:37:44.587978 3851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:37:44.588057 3851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:37:44.588120 3851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.67
I20260810 22:37:44.594574 3851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:38849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab
--principal=kudu/127.16.233.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.67:38143
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.233.67
--webserver_port=35907
--tserver_master_addrs=127.16.233.126:39115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.67
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:37:44.595738 3851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:37:44.596767 3851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:37:44.607381 3860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:37:44.607394 3858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:37:44.607525 3851 server_base.cc:1034] running on GCE node
W20260810 22:37:44.607385 3857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:37:44.608184 3851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:37:44.609004 3851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:37:44.610201 3851 hybrid_clock.cc:648] HybridClock initialized: now 1786401464610182 us; error 38 us; skew 500 ppm
Aug 10 22:37:44 dist-test-slave-77v9 krb5kdc[2204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401464, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:37:44.613308 3851 init.cc:377] Logged in from keytab as kudu/127.16.233.67@KRBTEST.COM (short username kudu)
I20260810 22:37:44.614324 3851 webserver.cc:460] Webserver started at http://127.16.233.67:35907/ using document root <none> and password file <none>
I20260810 22:37:44.614931 3851 fs_manager.cc:359] Metadata directory not provided
I20260810 22:37:44.614997 3851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:37:44.619136 3851 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 22:37:44.622064 3869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:37:44.624269 3851 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260810 22:37:44.624428 3851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-2/wal
uuid: "f1e19b4ab0404fbeb1208e06ed4bf0ee"
format_stamp: "Formatted at 2026-08-10 22:37:02 on dist-test-slave-77v9"
server_key: "b8e0bfe2c8721b4377d9f951121fd9d6"
server_key_iv: "62ce57aae310ed966623d263463c8469"
server_key_version: "encryptionkey@0"
I20260810 22:37:44.625346 3851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:37:44.642700 3851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:37:44.646884 3851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:37:44.647099 3851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:37:44.647933 3851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:37:44.649065 3851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:37:44.649129 3851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:37:44.649219 3851 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:37:44.649269 3851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:37:44.661439 3851 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.67:38143
I20260810 22:37:44.661501 3998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.67:38143 every 8 connection(s)
I20260810 22:37:44.662638 3851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
I20260810 22:37:44.663988 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 3851
I20260810 22:37:44.667124 4002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:44.665204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39115 (local address 127.16.233.67:38333)
0810 22:37:44.665748 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0810 22:37:44.665872 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":129,"threads_started":1}
W20260810 22:37:44.667649 3999 heartbeater.cc:644] Failed to heartbeat to 127.16.233.126:39115 (0 consecutive failures): Network error: Failed to ping master at 127.16.233.126:39115: Client connection negotiation failed: client connection to 127.16.233.126:39115: connect: Connection refused (error 111)
I20260810 22:37:44.668275 4002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:44.668047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39115 (local address 127.16.233.67:46891)
0810 22:37:44.668168 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0810 22:37:44.668217 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260810 22:37:44.668983 4002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:44.668725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39115 (local address 127.16.233.67:55549)
0810 22:37:44.668888 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0810 22:37:44.668934 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 22:37:44.669176 3999 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 22:37:45.355440 4003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:45.354907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39115 (local address 127.16.233.65:49597)
0810 22:37:45.355240 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0810 22:37:45.355309 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":90,"threads_started":1}
I20260810 22:37:45.463573 4004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:45.463039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39115 (local address 127.16.233.66:55501)
0810 22:37:45.463353 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0810 22:37:45.463438 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":84,"threads_started":1}
I20260810 22:37:45.510593 4005 raft_consensus.cc:482] T 9a2cb51d98444688a200c5d0278fcbb5 P f7df0146ce9c40c9a2d24d9834a2125f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:37:45.510866 4005 raft_consensus.cc:504] T 9a2cb51d98444688a200c5d0278fcbb5 P f7df0146ce9c40c9a2d24d9834a2125f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e19b4ab0404fbeb1208e06ed4bf0ee" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 38143 } } peers { permanent_uuid: "32bc0a72e0f7471186c4b04c2e8a0422" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 34433 } } peers { permanent_uuid: "f7df0146ce9c40c9a2d24d9834a2125f" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 36229 } }
I20260810 22:37:45.512152 4005 leader_election.cc:288] T 9a2cb51d98444688a200c5d0278fcbb5 P f7df0146ce9c40c9a2d24d9834a2125f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f1e19b4ab0404fbeb1208e06ed4bf0ee (127.16.233.67:38143), 32bc0a72e0f7471186c4b04c2e8a0422 (127.16.233.66:34433)
Aug 10 22:37:45 dist-test-slave-77v9 krb5kdc[2204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401464, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.65@KRBTEST.COM for kudu/127.16.233.66@KRBTEST.COM
Aug 10 22:37:45 dist-test-slave-77v9 krb5kdc[2204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401464, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.65@KRBTEST.COM for kudu/127.16.233.67@KRBTEST.COM
I20260810 22:37:45.531090 4003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:45.512568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.67:38143 (local address 127.16.233.65:50313)
0810 22:37:45.512984 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0810 22:37:45.513002 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 22:37:45.513128 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:37:45.515251 (+ 2123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:37:45.515258 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:37:45.516010 (+ 752us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:37:45.516930 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:37:45.516957 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:37:45.519057 (+ 2100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:37:45.519061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:37:45.520852 (+ 1791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:37:45.520867 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:37:45.521993 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:37:45.522001 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:37:45.522382 (+ 381us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:37:45.523289 (+ 907us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:37:45.523302 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:37:45.526019 (+ 2717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:37:45.528470 (+ 2451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:37:45.528472 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:37:45.528474 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:37:45.528674 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:37:45.529129 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:37:45.529132 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:37:45.529133 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:37:45.529259 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:37:45.530069 (+ 810us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:37:45.530072 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:37:45.530287 (+ 215us) client_negotiation.cc:769] Sending connection context
0810 22:37:45.530480 (+ 193us) client_negotiation.cc:240] Negotiation successful
0810 22:37:45.530888 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":449,"spinlock_wait_cycles":2688}
I20260810 22:37:45.531114 4008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:45.512769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:39939 (local address 127.16.233.66:34433)
0810 22:37:45.513260 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0810 22:37:45.513283 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0810 22:37:45.513326 (+ 43us) server_negotiation.cc:407] Connection header received
0810 22:37:45.514260 (+ 934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:37:45.514269 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:37:45.514753 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:37:45.515118 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:37:45.517164 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:37:45.518837 (+ 1673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:37:45.521017 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:37:45.521797 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:37:45.522008 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:37:45.526024 (+ 4016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:37:45.526030 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:37:45.526042 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:37:45.526075 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:37:45.528013 (+ 1938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:37:45.528782 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:37:45.528790 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:37:45.528796 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:37:45.528980 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:37:45.529408 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:37:45.529411 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:37:45.529412 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:37:45.529920 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:37:45.530068 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 22:37:45.530615 (+ 547us) server_negotiation.cc:299] Negotiation successful
0810 22:37:45.530897 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":146,"threads_started":1}
I20260810 22:37:45.531136 4006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:45.512568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.66:34433 (local address 127.16.233.65:39939)
0810 22:37:45.512984 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0810 22:37:45.513002 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 22:37:45.513158 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:37:45.515251 (+ 2093us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:37:45.515258 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:37:45.516010 (+ 752us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:37:45.516930 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:37:45.516957 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:37:45.519061 (+ 2104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:37:45.519064 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:37:45.520852 (+ 1788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:37:45.520867 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:37:45.521993 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:37:45.522001 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:37:45.522403 (+ 402us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:37:45.523200 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:37:45.523240 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:37:45.525743 (+ 2503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:37:45.528161 (+ 2418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:37:45.528169 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:37:45.528186 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:37:45.528625 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:37:45.529088 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:37:45.529091 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:37:45.529093 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:37:45.529265 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:37:45.530039 (+ 774us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:37:45.530046 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:37:45.530299 (+ 253us) client_negotiation.cc:769] Sending connection context
0810 22:37:45.530479 (+ 180us) client_negotiation.cc:240] Negotiation successful
0810 22:37:45.530968 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":604,"spinlock_wait_cycles":80512,"thread_start_us":83,"threads_started":1}
I20260810 22:37:45.531139 4007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:45.512769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:50313 (local address 127.16.233.67:38143)
0810 22:37:45.513260 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0810 22:37:45.513278 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 22:37:45.513326 (+ 48us) server_negotiation.cc:407] Connection header received
0810 22:37:45.514260 (+ 934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:37:45.514269 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:37:45.514753 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:37:45.515118 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:37:45.517165 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:37:45.518837 (+ 1672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:37:45.521016 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:37:45.521796 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:37:45.522025 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:37:45.526235 (+ 4210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:37:45.526240 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:37:45.526251 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:37:45.526283 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:37:45.528342 (+ 2059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:37:45.528840 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:37:45.528849 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:37:45.528855 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:37:45.528999 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:37:45.529371 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:37:45.529375 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:37:45.529377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:37:45.529942 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:37:45.530087 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 22:37:45.530628 (+ 541us) server_negotiation.cc:299] Negotiation successful
0810 22:37:45.530942 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":96,"threads_started":1}
W20260810 22:37:45.533816 3556 leader_election.cc:341] T 9a2cb51d98444688a200c5d0278fcbb5 P f7df0146ce9c40c9a2d24d9834a2125f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 32bc0a72e0f7471186c4b04c2e8a0422 (127.16.233.66:34433): Not found: Tablet not found: 9a2cb51d98444688a200c5d0278fcbb5
W20260810 22:37:45.534016 3558 leader_election.cc:341] T 9a2cb51d98444688a200c5d0278fcbb5 P f7df0146ce9c40c9a2d24d9834a2125f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f1e19b4ab0404fbeb1208e06ed4bf0ee (127.16.233.67:38143): Not found: Tablet not found: 9a2cb51d98444688a200c5d0278fcbb5
I20260810 22:37:45.534073 3556 leader_election.cc:302] T 9a2cb51d98444688a200c5d0278fcbb5 P f7df0146ce9c40c9a2d24d9834a2125f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7df0146ce9c40c9a2d24d9834a2125f; no voters: 32bc0a72e0f7471186c4b04c2e8a0422, f1e19b4ab0404fbeb1208e06ed4bf0ee
I20260810 22:37:45.534413 4005 raft_consensus.cc:2726] T 9a2cb51d98444688a200c5d0278fcbb5 P f7df0146ce9c40c9a2d24d9834a2125f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 22:37:45.670426 4009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:45.669927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39115 (local address 127.16.233.67:41269)
0810 22:37:45.670251 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0810 22:37:45.670318 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":104,"threads_started":1}
I20260810 22:37:46.357101 4010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:46.356444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39115 (local address 127.16.233.65:37997)
0810 22:37:46.356889 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0810 22:37:46.356982 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":92,"threads_started":1}
I20260810 22:37:46.465157 4011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:46.464619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39115 (local address 127.16.233.66:43467)
0810 22:37:46.464983 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0810 22:37:46.465051 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":212,"threads_started":1}
I20260810 22:37:46.671868 4012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:46.671378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39115 (local address 127.16.233.67:43709)
0810 22:37:46.671679 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0810 22:37:46.671746 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":87,"threads_started":1}
I20260810 22:37:47.358613 4013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:47.358093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39115 (local address 127.16.233.65:45185)
0810 22:37:47.358431 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0810 22:37:47.358504 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":112,"threads_started":1}
I20260810 22:37:47.467044 4014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:47.466265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39115 (local address 127.16.233.66:52343)
0810 22:37:47.466761 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0810 22:37:47.466925 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":154,"threads_started":1}
I20260810 22:37:47.655771 4015 raft_consensus.cc:482] T 9a2cb51d98444688a200c5d0278fcbb5 P f7df0146ce9c40c9a2d24d9834a2125f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:37:47.655941 4015 raft_consensus.cc:504] T 9a2cb51d98444688a200c5d0278fcbb5 P f7df0146ce9c40c9a2d24d9834a2125f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e19b4ab0404fbeb1208e06ed4bf0ee" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 38143 } } peers { permanent_uuid: "32bc0a72e0f7471186c4b04c2e8a0422" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 34433 } } peers { permanent_uuid: "f7df0146ce9c40c9a2d24d9834a2125f" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 36229 } }
I20260810 22:37:47.656603 4015 leader_election.cc:288] T 9a2cb51d98444688a200c5d0278fcbb5 P f7df0146ce9c40c9a2d24d9834a2125f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f1e19b4ab0404fbeb1208e06ed4bf0ee (127.16.233.67:38143), 32bc0a72e0f7471186c4b04c2e8a0422 (127.16.233.66:34433)
W20260810 22:37:47.657646 3558 leader_election.cc:341] T 9a2cb51d98444688a200c5d0278fcbb5 P f7df0146ce9c40c9a2d24d9834a2125f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f1e19b4ab0404fbeb1208e06ed4bf0ee (127.16.233.67:38143): Not found: Tablet not found: 9a2cb51d98444688a200c5d0278fcbb5
W20260810 22:37:47.657923 3556 leader_election.cc:341] T 9a2cb51d98444688a200c5d0278fcbb5 P f7df0146ce9c40c9a2d24d9834a2125f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 32bc0a72e0f7471186c4b04c2e8a0422 (127.16.233.66:34433): Not found: Tablet not found: 9a2cb51d98444688a200c5d0278fcbb5
I20260810 22:37:47.658020 3556 leader_election.cc:302] T 9a2cb51d98444688a200c5d0278fcbb5 P f7df0146ce9c40c9a2d24d9834a2125f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7df0146ce9c40c9a2d24d9834a2125f; no voters: 32bc0a72e0f7471186c4b04c2e8a0422, f1e19b4ab0404fbeb1208e06ed4bf0ee
I20260810 22:37:47.658259 4015 raft_consensus.cc:2726] T 9a2cb51d98444688a200c5d0278fcbb5 P f7df0146ce9c40c9a2d24d9834a2125f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 22:37:47.673588 4017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:47.673020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39115 (local address 127.16.233.67:48109)
0810 22:37:47.673399 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0810 22:37:47.673477 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":119,"threads_started":1}
I20260810 22:37:48.360028 4018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:48.359575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39115 (local address 127.16.233.65:35347)
0810 22:37:48.359851 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0810 22:37:48.359924 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":84,"threads_started":1}
I20260810 22:37:48.468758 4019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:48.468221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39115 (local address 127.16.233.66:41431)
0810 22:37:48.468571 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0810 22:37:48.468652 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":95,"threads_started":1}
I20260810 22:37:48.674909 4020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:48.674446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39115 (local address 127.16.233.67:34635)
0810 22:37:48.674747 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0810 22:37:48.674808 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":110,"threads_started":1}
I20260810 22:37:49.160660 4022 raft_consensus.cc:482] T 9a2cb51d98444688a200c5d0278fcbb5 P f7df0146ce9c40c9a2d24d9834a2125f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:37:49.160877 4022 raft_consensus.cc:504] T 9a2cb51d98444688a200c5d0278fcbb5 P f7df0146ce9c40c9a2d24d9834a2125f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e19b4ab0404fbeb1208e06ed4bf0ee" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 38143 } } peers { permanent_uuid: "32bc0a72e0f7471186c4b04c2e8a0422" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 34433 } } peers { permanent_uuid: "f7df0146ce9c40c9a2d24d9834a2125f" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 36229 } }
I20260810 22:37:49.161401 4022 leader_election.cc:288] T 9a2cb51d98444688a200c5d0278fcbb5 P f7df0146ce9c40c9a2d24d9834a2125f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f1e19b4ab0404fbeb1208e06ed4bf0ee (127.16.233.67:38143), 32bc0a72e0f7471186c4b04c2e8a0422 (127.16.233.66:34433)
W20260810 22:37:49.162395 3558 leader_election.cc:341] T 9a2cb51d98444688a200c5d0278fcbb5 P f7df0146ce9c40c9a2d24d9834a2125f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f1e19b4ab0404fbeb1208e06ed4bf0ee (127.16.233.67:38143): Not found: Tablet not found: 9a2cb51d98444688a200c5d0278fcbb5
W20260810 22:37:49.162492 3556 leader_election.cc:341] T 9a2cb51d98444688a200c5d0278fcbb5 P f7df0146ce9c40c9a2d24d9834a2125f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 32bc0a72e0f7471186c4b04c2e8a0422 (127.16.233.66:34433): Not found: Tablet not found: 9a2cb51d98444688a200c5d0278fcbb5
I20260810 22:37:49.162613 3556 leader_election.cc:302] T 9a2cb51d98444688a200c5d0278fcbb5 P f7df0146ce9c40c9a2d24d9834a2125f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7df0146ce9c40c9a2d24d9834a2125f; no voters: 32bc0a72e0f7471186c4b04c2e8a0422, f1e19b4ab0404fbeb1208e06ed4bf0ee
I20260810 22:37:49.162819 4022 raft_consensus.cc:2726] T 9a2cb51d98444688a200c5d0278fcbb5 P f7df0146ce9c40c9a2d24d9834a2125f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 22:37:49.361645 4023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:49.361146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39115 (local address 127.16.233.65:60643)
0810 22:37:49.361472 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0810 22:37:49.361533 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":79,"threads_started":1}
I20260810 22:37:49.470448 4024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:49.469925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39115 (local address 127.16.233.66:36403)
0810 22:37:49.470254 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0810 22:37:49.470324 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":107,"threads_started":1}
I20260810 22:37:49.676301 4025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:49.675808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39115 (local address 127.16.233.67:39117)
0810 22:37:49.676131 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0810 22:37:49.676198 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":91,"threads_started":1}
I20260810 22:37:50.363130 4026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:50.362636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39115 (local address 127.16.233.65:54759)
0810 22:37:50.362963 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0810 22:37:50.363026 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":109,"threads_started":1}
I20260810 22:37:50.471987 4027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:50.471423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39115 (local address 127.16.233.66:35879)
0810 22:37:50.471779 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0810 22:37:50.471855 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":139,"threads_started":1}
I20260810 22:37:50.677925 4028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:50.677387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39115 (local address 127.16.233.67:55051)
0810 22:37:50.677695 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0810 22:37:50.677822 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":103,"threads_started":1}
I20260810 22:37:51.364609 4029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:51.364116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39115 (local address 127.16.233.65:50173)
0810 22:37:51.364433 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0810 22:37:51.364503 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":87,"threads_started":1}
I20260810 22:37:51.473572 4030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:51.472999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39115 (local address 127.16.233.66:58253)
0810 22:37:51.473396 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0810 22:37:51.473469 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":104,"threads_started":1}
I20260810 22:37:51.679476 4031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:51.678858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39115 (local address 127.16.233.67:45151)
0810 22:37:51.679274 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0810 22:37:51.679373 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":110,"threads_started":1}
I20260810 22:37:52.366043 4032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:52.365500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39115 (local address 127.16.233.65:51871)
0810 22:37:52.365818 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0810 22:37:52.365934 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":87,"threads_started":1}
I20260810 22:37:52.475157 4034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:52.474639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39115 (local address 127.16.233.66:44599)
0810 22:37:52.474984 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0810 22:37:52.475051 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":112,"threads_started":1}
I20260810 22:37:52.681061 4035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:52.680540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39115 (local address 127.16.233.67:58683)
0810 22:37:52.680888 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0810 22:37:52.680957 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":86,"threads_started":1}
I20260810 22:37:52.813732 4037 raft_consensus.cc:482] T 9a2cb51d98444688a200c5d0278fcbb5 P f7df0146ce9c40c9a2d24d9834a2125f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:37:52.813879 4037 raft_consensus.cc:504] T 9a2cb51d98444688a200c5d0278fcbb5 P f7df0146ce9c40c9a2d24d9834a2125f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e19b4ab0404fbeb1208e06ed4bf0ee" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 38143 } } peers { permanent_uuid: "32bc0a72e0f7471186c4b04c2e8a0422" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 34433 } } peers { permanent_uuid: "f7df0146ce9c40c9a2d24d9834a2125f" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 36229 } }
I20260810 22:37:52.814395 4037 leader_election.cc:288] T 9a2cb51d98444688a200c5d0278fcbb5 P f7df0146ce9c40c9a2d24d9834a2125f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f1e19b4ab0404fbeb1208e06ed4bf0ee (127.16.233.67:38143), 32bc0a72e0f7471186c4b04c2e8a0422 (127.16.233.66:34433)
W20260810 22:37:52.815362 3558 leader_election.cc:341] T 9a2cb51d98444688a200c5d0278fcbb5 P f7df0146ce9c40c9a2d24d9834a2125f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f1e19b4ab0404fbeb1208e06ed4bf0ee (127.16.233.67:38143): Not found: Tablet not found: 9a2cb51d98444688a200c5d0278fcbb5
W20260810 22:37:52.815474 3556 leader_election.cc:341] T 9a2cb51d98444688a200c5d0278fcbb5 P f7df0146ce9c40c9a2d24d9834a2125f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 32bc0a72e0f7471186c4b04c2e8a0422 (127.16.233.66:34433): Not found: Tablet not found: 9a2cb51d98444688a200c5d0278fcbb5
I20260810 22:37:52.815569 3556 leader_election.cc:302] T 9a2cb51d98444688a200c5d0278fcbb5 P f7df0146ce9c40c9a2d24d9834a2125f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7df0146ce9c40c9a2d24d9834a2125f; no voters: 32bc0a72e0f7471186c4b04c2e8a0422, f1e19b4ab0404fbeb1208e06ed4bf0ee
I20260810 22:37:52.815732 4037 raft_consensus.cc:2726] T 9a2cb51d98444688a200c5d0278fcbb5 P f7df0146ce9c40c9a2d24d9834a2125f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 22:37:53.367410 4038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:53.366901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39115 (local address 127.16.233.65:37123)
0810 22:37:53.367237 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0810 22:37:53.367301 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":86,"threads_started":1}
I20260810 22:37:53.476642 4039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:53.476072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39115 (local address 127.16.233.66:59561)
0810 22:37:53.476472 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0810 22:37:53.476534 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":107,"threads_started":1}
I20260810 22:37:53.682546 4040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:53.682009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39115 (local address 127.16.233.67:38675)
0810 22:37:53.682366 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0810 22:37:53.682434 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":106,"threads_started":1}
I20260810 22:37:54.369095 4041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:54.368522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39115 (local address 127.16.233.65:43263)
0810 22:37:54.368886 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0810 22:37:54.368979 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":123,"threads_started":1}
I20260810 22:37:54.478183 4042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:54.477657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39115 (local address 127.16.233.66:45693)
0810 22:37:54.478004 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0810 22:37:54.478081 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":91,"threads_started":1}
I20260810 22:37:54.683910 4043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:54.683456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39115 (local address 127.16.233.67:45739)
0810 22:37:54.683746 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0810 22:37:54.683808 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":98,"threads_started":1}
I20260810 22:37:55.370464 4044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:55.369987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39115 (local address 127.16.233.65:43171)
0810 22:37:55.370295 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0810 22:37:55.370363 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":82,"threads_started":1}
I20260810 22:37:55.479640 4046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:55.479139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39115 (local address 127.16.233.66:41549)
0810 22:37:55.479476 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0810 22:37:55.479542 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":102,"threads_started":1}
I20260810 22:37:55.685316 4047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:55.684872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39115 (local address 127.16.233.67:37371)
0810 22:37:55.685147 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0810 22:37:55.685214 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":81,"threads_started":1}
I20260810 22:37:56.371948 4048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:56.371445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39115 (local address 127.16.233.65:44451)
0810 22:37:56.371774 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0810 22:37:56.371846 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":97,"threads_started":1}
I20260810 22:37:56.481230 4049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:56.480638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39115 (local address 127.16.233.66:43577)
0810 22:37:56.481059 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0810 22:37:56.481125 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":118,"threads_started":1}
I20260810 22:37:56.686771 4050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:56.686309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39115 (local address 127.16.233.67:59413)
0810 22:37:56.686607 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0810 22:37:56.686672 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":117,"threads_started":1}
I20260810 22:37:56.738140 4051 raft_consensus.cc:482] T 9a2cb51d98444688a200c5d0278fcbb5 P f7df0146ce9c40c9a2d24d9834a2125f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:37:56.738291 4051 raft_consensus.cc:504] T 9a2cb51d98444688a200c5d0278fcbb5 P f7df0146ce9c40c9a2d24d9834a2125f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e19b4ab0404fbeb1208e06ed4bf0ee" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 38143 } } peers { permanent_uuid: "32bc0a72e0f7471186c4b04c2e8a0422" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 34433 } } peers { permanent_uuid: "f7df0146ce9c40c9a2d24d9834a2125f" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 36229 } }
I20260810 22:37:56.738720 4051 leader_election.cc:288] T 9a2cb51d98444688a200c5d0278fcbb5 P f7df0146ce9c40c9a2d24d9834a2125f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f1e19b4ab0404fbeb1208e06ed4bf0ee (127.16.233.67:38143), 32bc0a72e0f7471186c4b04c2e8a0422 (127.16.233.66:34433)
W20260810 22:37:56.739629 3558 leader_election.cc:341] T 9a2cb51d98444688a200c5d0278fcbb5 P f7df0146ce9c40c9a2d24d9834a2125f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f1e19b4ab0404fbeb1208e06ed4bf0ee (127.16.233.67:38143): Not found: Tablet not found: 9a2cb51d98444688a200c5d0278fcbb5
W20260810 22:37:56.739890 3556 leader_election.cc:341] T 9a2cb51d98444688a200c5d0278fcbb5 P f7df0146ce9c40c9a2d24d9834a2125f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 32bc0a72e0f7471186c4b04c2e8a0422 (127.16.233.66:34433): Not found: Tablet not found: 9a2cb51d98444688a200c5d0278fcbb5
I20260810 22:37:56.739984 3556 leader_election.cc:302] T 9a2cb51d98444688a200c5d0278fcbb5 P f7df0146ce9c40c9a2d24d9834a2125f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7df0146ce9c40c9a2d24d9834a2125f; no voters: 32bc0a72e0f7471186c4b04c2e8a0422, f1e19b4ab0404fbeb1208e06ed4bf0ee
I20260810 22:37:56.740187 4051 raft_consensus.cc:2726] T 9a2cb51d98444688a200c5d0278fcbb5 P f7df0146ce9c40c9a2d24d9834a2125f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 22:37:57.373502 4052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:57.373022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39115 (local address 127.16.233.65:41419)
0810 22:37:57.373321 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0810 22:37:57.373377 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":102,"threads_started":1}
I20260810 22:37:57.482770 4053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:57.482254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39115 (local address 127.16.233.66:37627)
0810 22:37:57.482603 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0810 22:37:57.482669 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":113,"threads_started":1}
I20260810 22:37:57.688256 4054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:57.687783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39115 (local address 127.16.233.67:45981)
0810 22:37:57.688092 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0810 22:37:57.688153 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":98,"threads_started":1}
I20260810 22:37:58.374951 4055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:58.374441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39115 (local address 127.16.233.65:46599)
0810 22:37:58.374776 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0810 22:37:58.374846 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":96,"threads_started":1}
I20260810 22:37:58.484150 4056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:58.483685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39115 (local address 127.16.233.66:58641)
0810 22:37:58.483983 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0810 22:37:58.484048 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":108,"threads_started":1}
I20260810 22:37:58.690028 4057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:58.689380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39115 (local address 127.16.233.67:57411)
0810 22:37:58.689709 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0810 22:37:58.689857 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":93,"threads_started":1}
I20260810 22:37:59.376724 4058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:59.376191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39115 (local address 127.16.233.65:55451)
0810 22:37:59.376542 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0810 22:37:59.376619 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":120,"threads_started":1}
I20260810 22:37:59.486136 4059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:37:59.485598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:39115 (local address 127.16.233.66:54475)
0810 22:37:59.485964 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0810 22:37:59.486033 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.126:39115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":81,"threads_started":1}
I20260810 22:37:59.666375 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 3492
I20260810 22:37:59.675959 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 3694
I20260810 22:37:59.685639 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 3851
2026-08-10T22:37:59Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59681 ms)
[----------] 30 tests from SecurityITest (264244 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:37:59 dist-test-slave-77v9 krb5kdc[4063](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:37:59 dist-test-slave-77v9 krb5kdc[4063](info): set up 2 sockets
Aug 10 22:37:59 dist-test-slave-77v9 krb5kdc[4063](info): commencing operation
krb5kdc: starting...
W20260810 22:38:00.858482 17317 mini_kdc.cc:121] Time spent starting KDC: real 1.145s user 0.003s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Aug 10 22:38:00 dist-test-slave-77v9 krb5kdc[4063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:38:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:38:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.233.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab.
Entry for principal kudu/127.16.233.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab.
I20260810 22:38:01.008631 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:37427
--webserver_interface=127.16.233.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab
--principal=kudu/127.16.233.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:45013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.126:37427
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:38:01.128991 4080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:38:01.129309 4080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:38:01.129352 4080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:38:01.133509 4080 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:38:01.133582 4080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:38:01.133602 4080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:38:01.133620 4080 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:38:01.133635 4080 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:38:01.139190 4080 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:45013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.126:37427
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab
--principal=kudu/127.16.233.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:37427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:38:01.140364 4080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:38:01.141425 4080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:38:01.150913 4088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:01.150920 4086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:38:01.151273 4080 server_base.cc:1034] running on GCE node
W20260810 22:38:01.150969 4085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:38:01.152117 4080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:38:01.153487 4080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:38:01.154668 4080 hybrid_clock.cc:648] HybridClock initialized: now 1786401481154649 us; error 34 us; skew 500 ppm
Aug 10 22:38:01 dist-test-slave-77v9 krb5kdc[4063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401481, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:38:01.157833 4080 init.cc:377] Logged in from keytab as kudu/127.16.233.126@KRBTEST.COM (short username kudu)
I20260810 22:38:01.158887 4080 webserver.cc:460] Webserver started at http://127.16.233.126:44819/ using document root <none> and password file <none>
I20260810 22:38:01.159498 4080 fs_manager.cc:359] Metadata directory not provided
I20260810 22:38:01.159597 4080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:38:01.159850 4080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:38:01.162266 4080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/minicluster-data/master-0/data/instance:
uuid: "5ae784d29260410b8c807b3121b7671e"
format_stamp: "Formatted at 2026-08-10 22:38:01 on dist-test-slave-77v9"
server_key: "c0ab0e291f076af345ed9f8d029194c2"
server_key_iv: "222ede96af71654ef6cfa926ea14e126"
server_key_version: "encryptionkey@0"
I20260810 22:38:01.163017 4080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/minicluster-data/master-0/wal/instance:
uuid: "5ae784d29260410b8c807b3121b7671e"
format_stamp: "Formatted at 2026-08-10 22:38:01 on dist-test-slave-77v9"
server_key: "c0ab0e291f076af345ed9f8d029194c2"
server_key_iv: "222ede96af71654ef6cfa926ea14e126"
server_key_version: "encryptionkey@0"
I20260810 22:38:01.167441 4080 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 22:38:01.170275 4097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:01.171804 4080 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 22:38:01.172043 4080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/minicluster-data/master-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/minicluster-data/master-0/wal
uuid: "5ae784d29260410b8c807b3121b7671e"
format_stamp: "Formatted at 2026-08-10 22:38:01 on dist-test-slave-77v9"
server_key: "c0ab0e291f076af345ed9f8d029194c2"
server_key_iv: "222ede96af71654ef6cfa926ea14e126"
server_key_version: "encryptionkey@0"
I20260810 22:38:01.172210 4080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:38:01.185770 4080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:38:01.189553 4080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:38:01.189843 4080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:38:01.198869 4080 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.126:37427
I20260810 22:38:01.198904 4160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.126:37427 every 8 connection(s)
I20260810 22:38:01.200043 4080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
I20260810 22:38:01.203166 4161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:38:01.207497 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 4080
I20260810 22:38:01.207631 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/minicluster-data/master-0/wal/instance
I20260810 22:38:01.208020 17317 external_mini_cluster.cc:1428] Setting key ea812403352d40d96fc7b5a728bbbee8
I20260810 22:38:01.209772 4161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ae784d29260410b8c807b3121b7671e: Bootstrap starting.
I20260810 22:38:01.212406 4161 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ae784d29260410b8c807b3121b7671e: Neither blocks nor log segments found. Creating new log.
I20260810 22:38:01.213337 4161 log.cc:824] T 00000000000000000000000000000000 P 5ae784d29260410b8c807b3121b7671e: Log is configured to *not* fsync() on all Append() calls
I20260810 22:38:01.216048 4161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ae784d29260410b8c807b3121b7671e: No bootstrap required, opened a new log
Aug 10 22:38:01 dist-test-slave-77v9 krb5kdc[4063](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:38:01.221804 4161 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ae784d29260410b8c807b3121b7671e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ae784d29260410b8c807b3121b7671e" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 37427 } }
I20260810 22:38:01.222182 4161 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ae784d29260410b8c807b3121b7671e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:38:01.222241 4161 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ae784d29260410b8c807b3121b7671e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ae784d29260410b8c807b3121b7671e, State: Initialized, Role: FOLLOWER
I20260810 22:38:01.222898 4161 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ae784d29260410b8c807b3121b7671e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ae784d29260410b8c807b3121b7671e" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 37427 } }
I20260810 22:38:01.223284 4161 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ae784d29260410b8c807b3121b7671e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:38:01.223397 4161 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ae784d29260410b8c807b3121b7671e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:38:01.223537 4161 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ae784d29260410b8c807b3121b7671e [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:38:01.224696 4161 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ae784d29260410b8c807b3121b7671e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ae784d29260410b8c807b3121b7671e" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 37427 } }
I20260810 22:38:01.225168 4161 leader_election.cc:302] T 00000000000000000000000000000000 P 5ae784d29260410b8c807b3121b7671e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ae784d29260410b8c807b3121b7671e; no voters:
I20260810 22:38:01.225548 4161 leader_election.cc:288] T 00000000000000000000000000000000 P 5ae784d29260410b8c807b3121b7671e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:38:01.225672 4166 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ae784d29260410b8c807b3121b7671e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:38:01.225903 4166 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ae784d29260410b8c807b3121b7671e [term 1 LEADER]: Becoming Leader. State: Replica: 5ae784d29260410b8c807b3121b7671e, State: Running, Role: LEADER
I20260810 22:38:01.226326 4164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:01.209667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41302 (local address 127.16.233.126:37427)
0810 22:38:01.210206 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:01.210223 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:01.210264 (+ 41us) server_negotiation.cc:407] Connection header received
0810 22:38:01.211013 (+ 749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:01.211031 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:01.211433 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:01.211806 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:01.212781 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:01.214007 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:01.215339 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:01.215888 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:01.216014 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:01.219607 (+ 3593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:01.219618 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:01.219649 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:38:01.219705 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:01.222181 (+ 2476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:01.223006 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:01.223017 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:01.223024 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:01.223237 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:01.223697 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:01.223700 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:01.223702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:01.224194 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:01.224463 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:01.224731 (+ 268us) server_negotiation.cc:299] Negotiation successful
0810 22:38:01.225149 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":174,"threads_started":1}
I20260810 22:38:01.226325 4166 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ae784d29260410b8c807b3121b7671e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ae784d29260410b8c807b3121b7671e" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 37427 } }
I20260810 22:38:01.227113 4161 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ae784d29260410b8c807b3121b7671e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:38:01.229408 4167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ae784d29260410b8c807b3121b7671e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ae784d29260410b8c807b3121b7671e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ae784d29260410b8c807b3121b7671e" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 37427 } } }
I20260810 22:38:01.229558 4167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ae784d29260410b8c807b3121b7671e [sys.catalog]: This master's current role is: LEADER
I20260810 22:38:01.229957 4168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ae784d29260410b8c807b3121b7671e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ae784d29260410b8c807b3121b7671e. Latest consensus state: current_term: 1 leader_uuid: "5ae784d29260410b8c807b3121b7671e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ae784d29260410b8c807b3121b7671e" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 37427 } } }
I20260810 22:38:01.230083 4168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ae784d29260410b8c807b3121b7671e [sys.catalog]: This master's current role is: LEADER
W20260810 22:38:01.232012 4183 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5ae784d29260410b8c807b3121b7671e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 22:38:01.232087 4183 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 22:38:01.232177 4184 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:38:01.233014 4184 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:38:01.239603 4184 catalog_manager.cc:1347] Generated new cluster ID: 8db1447796aa44bfab5d58d6cffd25b4
I20260810 22:38:01.239711 4184 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:38:01.248414 4184 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:38:01.249898 4184 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:38:01.261324 4184 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ae784d29260410b8c807b3121b7671e: Generated new TSK 0
I20260810 22:38:01.262363 4184 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.233.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab.
Entry for principal kudu/127.16.233.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab.
I20260810 22:38:01.301364 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.65:0
--local_ip_for_outbound_sockets=127.16.233.65
--webserver_interface=127.16.233.65
--webserver_port=0
--tserver_master_addrs=127.16.233.126:37427
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab
--principal=kudu/127.16.233.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:45013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:38:01.424196 4191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:38:01.424522 4191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:38:01.424571 4191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:38:01.431669 4191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:38:01.431777 4191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:38:01.431870 4191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.65
I20260810 22:38:01.439533 4191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:45013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab
--principal=kudu/127.16.233.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.233.65
--webserver_port=0
--tserver_master_addrs=127.16.233.126:37427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.65
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:38:01.440685 4191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:38:01.441807 4191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:38:01.452729 4199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:01.452662 4197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:01.452662 4196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:38:01.452782 4191 server_base.cc:1034] running on GCE node
I20260810 22:38:01.453491 4191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:38:01.454166 4191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:38:01.455369 4191 hybrid_clock.cc:648] HybridClock initialized: now 1786401481455356 us; error 29 us; skew 500 ppm
Aug 10 22:38:01 dist-test-slave-77v9 krb5kdc[4063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401481, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:38:01.458437 4191 init.cc:377] Logged in from keytab as kudu/127.16.233.65@KRBTEST.COM (short username kudu)
I20260810 22:38:01.459511 4191 webserver.cc:460] Webserver started at http://127.16.233.65:45917/ using document root <none> and password file <none>
I20260810 22:38:01.460129 4191 fs_manager.cc:359] Metadata directory not provided
I20260810 22:38:01.460232 4191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:38:01.460481 4191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:38:01.462782 4191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/minicluster-data/ts-0/data/instance:
uuid: "732a0e621f7e473ea16149e69b7568a9"
format_stamp: "Formatted at 2026-08-10 22:38:01 on dist-test-slave-77v9"
server_key: "54a55ae3f2523edf8c7396c50aefb011"
server_key_iv: "af63bba9cd509f397635a9453f89c8d6"
server_key_version: "encryptionkey@0"
I20260810 22:38:01.463413 4191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/minicluster-data/ts-0/wal/instance:
uuid: "732a0e621f7e473ea16149e69b7568a9"
format_stamp: "Formatted at 2026-08-10 22:38:01 on dist-test-slave-77v9"
server_key: "54a55ae3f2523edf8c7396c50aefb011"
server_key_iv: "af63bba9cd509f397635a9453f89c8d6"
server_key_version: "encryptionkey@0"
I20260810 22:38:01.468154 4191 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 22:38:01.471195 4208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:01.472630 4191 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 22:38:01.472798 4191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/minicluster-data/ts-0/wal
uuid: "732a0e621f7e473ea16149e69b7568a9"
format_stamp: "Formatted at 2026-08-10 22:38:01 on dist-test-slave-77v9"
server_key: "54a55ae3f2523edf8c7396c50aefb011"
server_key_iv: "af63bba9cd509f397635a9453f89c8d6"
server_key_version: "encryptionkey@0"
I20260810 22:38:01.473012 4191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:38:01.491871 4191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:38:01.495814 4191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:38:01.496066 4191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:38:01.496919 4191 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:38:01.498003 4191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:38:01.498086 4191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:01.498162 4191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:38:01.498196 4191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:01.509312 4191 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.65:38629
I20260810 22:38:01.509348 4332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.65:38629 every 8 connection(s)
I20260810 22:38:01.510423 4191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
I20260810 22:38:01.510672 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 4191
I20260810 22:38:01.510826 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/minicluster-data/ts-0/wal/instance
I20260810 22:38:01.511214 17317 external_mini_cluster.cc:1428] Setting key 7e8f70c9d87814f5a659bcef20c59a3b
Aug 10 22:38:01 dist-test-slave-77v9 krb5kdc[4063](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401481, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.65@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.233.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.66@KRBTEST.COM" created.
I20260810 22:38:01.529670 4164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:01.513001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:45505 (local address 127.16.233.126:37427)
0810 22:38:01.513157 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:01.513165 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:01.514456 (+ 1291us) server_negotiation.cc:407] Connection header received
0810 22:38:01.516432 (+ 1976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:01.516439 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:01.516514 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:01.516627 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:01.518320 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:01.518968 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:01.521029 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:01.521272 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:01.521382 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:01.524600 (+ 3218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:01.524603 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:01.524606 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:38:01.524634 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:01.526427 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:01.527401 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:01.527408 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:01.527413 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:01.527521 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:01.528040 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:01.528044 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:01.528045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:01.528364 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:01.528453 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:01.529369 (+ 916us) server_negotiation.cc:299] Negotiation successful
0810 22:38:01.529522 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 22:38:01.530799 4335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:01.513627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:37427 (local address 127.16.233.65:45505)
0810 22:38:01.514246 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:01.514313 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0810 22:38:01.516148 (+ 1835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:38:01.516906 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:38:01.516914 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:38:01.517301 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:38:01.518097 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:01.518120 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:01.519125 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:01.519132 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:01.520843 (+ 1711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:01.520859 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:01.521471 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:01.521476 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:01.521706 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:01.522455 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:38:01.522483 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:38:01.524389 (+ 1906us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:38:01.526587 (+ 2198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:38:01.526593 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:38:01.526608 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:38:01.527222 (+ 614us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:38:01.527658 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:38:01.527662 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:38:01.527664 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:38:01.527907 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:38:01.528487 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:38:01.528497 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:38:01.528947 (+ 450us) client_negotiation.cc:769] Sending connection context
0810 22:38:01.529197 (+ 250us) client_negotiation.cc:240] Negotiation successful
0810 22:38:01.529700 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":151,"threads_started":1}
I20260810 22:38:01.532274 4333 heartbeater.cc:347] Connected to a master server at 127.16.233.126:37427
I20260810 22:38:01.532559 4333 heartbeater.cc:464] Registering TS with master...
I20260810 22:38:01.533182 4333 heartbeater.cc:511] Master 127.16.233.126:37427 requested a full tablet report, sending...
I20260810 22:38:01.535035 4125 ts_manager.cc:194] Registered new tserver with Master: 732a0e621f7e473ea16149e69b7568a9 (127.16.233.65:38629)
I20260810 22:38:01.536912 4125 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.65@KRBTEST.COM'} at 127.16.233.65:45505
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab.
Entry for principal kudu/127.16.233.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab.
I20260810 22:38:01.549023 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.66:0
--local_ip_for_outbound_sockets=127.16.233.66
--webserver_interface=127.16.233.66
--webserver_port=0
--tserver_master_addrs=127.16.233.126:37427
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab
--principal=kudu/127.16.233.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:45013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:38:01.668799 4338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:38:01.669147 4338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:38:01.669195 4338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:38:01.673498 4338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:38:01.673573 4338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:38:01.673671 4338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.66
I20260810 22:38:01.679639 4338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:45013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab
--principal=kudu/127.16.233.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.233.66
--webserver_port=0
--tserver_master_addrs=127.16.233.126:37427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.66
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:38:01.680948 4338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:38:01.682123 4338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:38:01.694419 4347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:01.694419 4345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:38:01.694450 4338 server_base.cc:1034] running on GCE node
W20260810 22:38:01.694422 4344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:38:01.695220 4338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:38:01.695952 4338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:38:01.697177 4338 hybrid_clock.cc:648] HybridClock initialized: now 1786401481697147 us; error 32 us; skew 500 ppm
Aug 10 22:38:01 dist-test-slave-77v9 krb5kdc[4063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401481, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:38:01.700449 4338 init.cc:377] Logged in from keytab as kudu/127.16.233.66@KRBTEST.COM (short username kudu)
I20260810 22:38:01.701592 4338 webserver.cc:460] Webserver started at http://127.16.233.66:33383/ using document root <none> and password file <none>
I20260810 22:38:01.702231 4338 fs_manager.cc:359] Metadata directory not provided
I20260810 22:38:01.702363 4338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:38:01.702620 4338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:38:01.704989 4338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/minicluster-data/ts-1/data/instance:
uuid: "28ffb60cc303403f950a133672dc0ae6"
format_stamp: "Formatted at 2026-08-10 22:38:01 on dist-test-slave-77v9"
server_key: "e1a0f578d0e30b5e97a9554b6077cb6f"
server_key_iv: "8021930c0ed1845f2fb4ba1e25f601f9"
server_key_version: "encryptionkey@0"
I20260810 22:38:01.705616 4338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/minicluster-data/ts-1/wal/instance:
uuid: "28ffb60cc303403f950a133672dc0ae6"
format_stamp: "Formatted at 2026-08-10 22:38:01 on dist-test-slave-77v9"
server_key: "e1a0f578d0e30b5e97a9554b6077cb6f"
server_key_iv: "8021930c0ed1845f2fb4ba1e25f601f9"
server_key_version: "encryptionkey@0"
I20260810 22:38:01.709887 4338 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 22:38:01.712621 4355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:01.714191 4338 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 22:38:01.714375 4338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/minicluster-data/ts-1/wal
uuid: "28ffb60cc303403f950a133672dc0ae6"
format_stamp: "Formatted at 2026-08-10 22:38:01 on dist-test-slave-77v9"
server_key: "e1a0f578d0e30b5e97a9554b6077cb6f"
server_key_iv: "8021930c0ed1845f2fb4ba1e25f601f9"
server_key_version: "encryptionkey@0"
I20260810 22:38:01.714530 4338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:38:01.731863 4338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:38:01.735558 4338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:38:01.735795 4338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:38:01.736501 4338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:38:01.737519 4338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:38:01.737588 4338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:01.737655 4338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:38:01.737702 4338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:01.747440 4338 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.66:44561
I20260810 22:38:01.747466 4490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.66:44561 every 8 connection(s)
I20260810 22:38:01.748564 4338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
I20260810 22:38:01.757951 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 4338
I20260810 22:38:01.758119 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/minicluster-data/ts-1/wal/instance
I20260810 22:38:01.758522 17317 external_mini_cluster.cc:1428] Setting key cb8adf52fac92174bd837f614a5de145
Aug 10 22:38:01 dist-test-slave-77v9 krb5kdc[4063](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401481, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.66@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:38:01.765038 4164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:01.750612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:48193 (local address 127.16.233.126:37427)
0810 22:38:01.750774 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:01.750778 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:01.751733 (+ 955us) server_negotiation.cc:407] Connection header received
0810 22:38:01.753384 (+ 1651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:01.753388 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:01.753457 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:01.753568 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:01.754851 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:01.755410 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:01.756634 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:01.756966 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:01.757080 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:01.760589 (+ 3509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:01.760593 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:01.760595 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:38:01.760622 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:01.762326 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:01.763011 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:01.763017 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:01.763019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:01.763126 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:01.763616 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:01.763622 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:01.763625 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:01.763903 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:01.764021 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:01.764592 (+ 571us) server_negotiation.cc:299] Negotiation successful
0810 22:38:01.764746 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 22:38:01.765931 4493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:01.751009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:37427 (local address 127.16.233.66:48193)
0810 22:38:01.751559 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:01.751607 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 22:38:01.753123 (+ 1516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:38:01.753762 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:38:01.753769 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:38:01.754171 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:38:01.754684 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:01.754699 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:01.755563 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:01.755567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:01.756487 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:01.756498 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:01.757150 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:01.757155 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:01.757333 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:01.758229 (+ 896us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:38:01.758250 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:38:01.760386 (+ 2136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:38:01.762478 (+ 2092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:38:01.762489 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:38:01.762513 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:38:01.762895 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:38:01.763298 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:38:01.763302 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:38:01.763304 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:38:01.763510 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:38:01.764039 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:38:01.764045 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:38:01.764320 (+ 275us) client_negotiation.cc:769] Sending connection context
0810 22:38:01.764499 (+ 179us) client_negotiation.cc:240] Negotiation successful
0810 22:38:01.764791 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":134,"threads_started":1}
I20260810 22:38:01.767290 4491 heartbeater.cc:347] Connected to a master server at 127.16.233.126:37427
I20260810 22:38:01.767609 4491 heartbeater.cc:464] Registering TS with master...
I20260810 22:38:01.768354 4491 heartbeater.cc:511] Master 127.16.233.126:37427 requested a full tablet report, sending...
I20260810 22:38:01.769718 4125 ts_manager.cc:194] Registered new tserver with Master: 28ffb60cc303403f950a133672dc0ae6 (127.16.233.66:44561)
WARNING: no policy specified for kudu/127.16.233.67@KRBTEST.COM; defaulting to no policy
I20260810 22:38:01.770380 4125 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.66@KRBTEST.COM'} at 127.16.233.66:48193
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab.
Entry for principal kudu/127.16.233.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab.
I20260810 22:38:01.793013 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.67:0
--local_ip_for_outbound_sockets=127.16.233.67
--webserver_interface=127.16.233.67
--webserver_port=0
--tserver_master_addrs=127.16.233.126:37427
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab
--principal=kudu/127.16.233.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:45013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:38:01.909854 4497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:38:01.910207 4497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:38:01.910288 4497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:38:01.914475 4497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:38:01.914597 4497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:38:01.914697 4497 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.67
I20260810 22:38:01.920536 4497 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:45013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab
--principal=kudu/127.16.233.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.233.67
--webserver_port=0
--tserver_master_addrs=127.16.233.126:37427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4497
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.67
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:38:01.921875 4497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:38:01.923454 4497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:38:01.934383 4503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:01.934499 4506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:01.934383 4504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:38:01.934677 4497 server_base.cc:1034] running on GCE node
I20260810 22:38:01.935154 4497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:38:01.935874 4497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:38:01.937040 4497 hybrid_clock.cc:648] HybridClock initialized: now 1786401481937024 us; error 32 us; skew 500 ppm
Aug 10 22:38:01 dist-test-slave-77v9 krb5kdc[4063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401481, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:38:01.940042 4497 init.cc:377] Logged in from keytab as kudu/127.16.233.67@KRBTEST.COM (short username kudu)
I20260810 22:38:01.941118 4497 webserver.cc:460] Webserver started at http://127.16.233.67:36673/ using document root <none> and password file <none>
I20260810 22:38:01.941687 4497 fs_manager.cc:359] Metadata directory not provided
I20260810 22:38:01.941752 4497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:38:01.941958 4497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:38:01.944242 4497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/minicluster-data/ts-2/data/instance:
uuid: "7d64a7f7068449bca9996c132073afb4"
format_stamp: "Formatted at 2026-08-10 22:38:01 on dist-test-slave-77v9"
server_key: "2ca35e2c455090e1d6bdd7b6ff504504"
server_key_iv: "afc0bba4f777e0075d9b1ac58a0d9b58"
server_key_version: "encryptionkey@0"
I20260810 22:38:01.944860 4497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/minicluster-data/ts-2/wal/instance:
uuid: "7d64a7f7068449bca9996c132073afb4"
format_stamp: "Formatted at 2026-08-10 22:38:01 on dist-test-slave-77v9"
server_key: "2ca35e2c455090e1d6bdd7b6ff504504"
server_key_iv: "afc0bba4f777e0075d9b1ac58a0d9b58"
server_key_version: "encryptionkey@0"
I20260810 22:38:01.949000 4497 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 22:38:01.951675 4514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:01.953260 4497 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:38:01.953413 4497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/minicluster-data/ts-2/wal
uuid: "7d64a7f7068449bca9996c132073afb4"
format_stamp: "Formatted at 2026-08-10 22:38:01 on dist-test-slave-77v9"
server_key: "2ca35e2c455090e1d6bdd7b6ff504504"
server_key_iv: "afc0bba4f777e0075d9b1ac58a0d9b58"
server_key_version: "encryptionkey@0"
I20260810 22:38:01.953544 4497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:38:01.972633 4497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:38:01.976410 4497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:38:01.976647 4497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:38:01.977557 4497 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:38:01.978523 4497 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:38:01.978611 4497 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:01.978682 4497 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:38:01.978731 4497 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:01.988265 4497 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.67:35873
I20260810 22:38:01.988445 4642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.67:35873 every 8 connection(s)
I20260810 22:38:01.989430 4497 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
I20260810 22:38:01.991684 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 4497
I20260810 22:38:01.991840 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/minicluster-data/ts-2/wal/instance
I20260810 22:38:01.992214 17317 external_mini_cluster.cc:1428] Setting key 068974066f7abacbfc97fd9cd57a6f2e
Aug 10 22:38:01 dist-test-slave-77v9 krb5kdc[4063](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401481, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.67@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:38:02.005378 4164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:01.991658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:42063 (local address 127.16.233.126:37427)
0810 22:38:01.991941 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:01.991945 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:01.993184 (+ 1239us) server_negotiation.cc:407] Connection header received
0810 22:38:01.994819 (+ 1635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:01.994822 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:01.994871 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:01.994959 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:01.996206 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:01.996770 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:01.998125 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:01.998352 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:01.998513 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:02.001466 (+ 2953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:02.001469 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:02.001472 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:38:02.001501 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:02.003103 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:02.003697 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:02.003699 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:02.003700 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:02.003784 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:02.004152 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:02.004154 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:02.004156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:02.004364 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:02.004446 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:02.005076 (+ 630us) server_negotiation.cc:299] Negotiation successful
0810 22:38:02.005200 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179}
I20260810 22:38:02.006495 4645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:01.992171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:37427 (local address 127.16.233.67:42063)
0810 22:38:01.993020 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:01.993071 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 22:38:01.994569 (+ 1498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:38:01.995153 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:38:01.995160 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:38:01.995547 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:38:01.996057 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:01.996072 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:01.996942 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:01.996945 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:01.997944 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:01.997956 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:01.998584 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:01.998590 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:01.998812 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:01.999551 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:38:01.999569 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:38:02.001248 (+ 1679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:38:02.003243 (+ 1995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:38:02.003250 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:38:02.003267 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:38:02.003598 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:38:02.003907 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:38:02.003909 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:38:02.003910 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:38:02.004064 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:38:02.004471 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:38:02.004477 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:38:02.004747 (+ 270us) client_negotiation.cc:769] Sending connection context
0810 22:38:02.004993 (+ 246us) client_negotiation.cc:240] Negotiation successful
0810 22:38:02.005410 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":195,"threads_started":1}
I20260810 22:38:02.007737 4643 heartbeater.cc:347] Connected to a master server at 127.16.233.126:37427
I20260810 22:38:02.008003 4643 heartbeater.cc:464] Registering TS with master...
I20260810 22:38:02.008533 4643 heartbeater.cc:511] Master 127.16.233.126:37427 requested a full tablet report, sending...
I20260810 22:38:02.009704 4125 ts_manager.cc:194] Registered new tserver with Master: 7d64a7f7068449bca9996c132073afb4 (127.16.233.67:35873)
I20260810 22:38:02.010419 4125 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.67@KRBTEST.COM'} at 127.16.233.67:42063
I20260810 22:38:02.019527 17317 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 22:38:02.030704 4164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:02.021982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41316 (local address 127.16.233.126:37427)
0810 22:38:02.022145 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:02.022150 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:02.022338 (+ 188us) server_negotiation.cc:407] Connection header received
0810 22:38:02.022446 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:02.022448 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:02.022496 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:02.022581 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:02.023547 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:02.024114 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:02.025253 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:02.025501 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:02.025598 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:02.027254 (+ 1656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:02.027258 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:02.027261 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:38:02.027289 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:02.028998 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:02.029434 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:02.029436 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:02.029437 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:02.029556 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:02.029843 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:02.029846 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:02.029847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:02.030089 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:02.030199 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:02.030463 (+ 264us) server_negotiation.cc:299] Negotiation successful
0810 22:38:02.030574 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 22:38:02.034092 4125 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41316:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 22:38:02.037271 4125 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:38:02.052690 4658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:02.046271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.67:35873 (local address 127.0.0.1:49832)
0810 22:38:02.047224 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:02.047281 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 22:38:02.047438 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:38:02.048226 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:38:02.048232 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:38:02.048255 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:38:02.048417 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:02.048424 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:02.049973 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:02.049976 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:02.051225 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:02.051236 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:02.052125 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:02.052128 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:02.052238 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:02.052276 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 22:38:02.052389 (+ 113us) client_negotiation.cc:240] Negotiation successful
0810 22:38:02.052521 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":764,"thread_start_us":145,"threads_started":1}
I20260810 22:38:02.052932 4660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:02.046595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40384 (local address 127.16.233.65:38629)
0810 22:38:02.047897 (+ 1302us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:02.047903 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:02.047942 (+ 39us) server_negotiation.cc:407] Connection header received
0810 22:38:02.048022 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:02.048027 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:02.048207 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:02.048355 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:38:02.048655 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:02.049611 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:02.051230 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:02.052109 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:02.052224 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:02.052387 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:02.052613 (+ 226us) server_negotiation.cc:299] Negotiation successful
0810 22:38:02.052720 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1192,"thread_start_us":104,"threads_started":1}
I20260810 22:38:02.053061 4659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:02.046423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49832 (local address 127.16.233.67:35873)
0810 22:38:02.047687 (+ 1264us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:02.047701 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:02.047726 (+ 25us) server_negotiation.cc:407] Connection header received
0810 22:38:02.047823 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:02.047829 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:02.048068 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:02.048240 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:38:02.048552 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:02.049808 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:02.051403 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:02.051990 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:02.052122 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:02.052283 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:02.052791 (+ 508us) server_negotiation.cc:299] Negotiation successful
0810 22:38:02.052934 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1148,"thread_start_us":161,"threads_started":1}
I20260810 22:38:02.052708 4656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:02.046333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.65:38629 (local address 127.0.0.1:40384)
0810 22:38:02.047411 (+ 1078us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:02.047426 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:38:02.047510 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:38:02.048414 (+ 904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:38:02.048417 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:38:02.048432 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:38:02.048532 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:02.048538 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:02.049726 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:02.049731 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:02.051102 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:02.051112 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:02.052379 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:02.052384 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:02.052466 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:02.052486 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 22:38:02.052558 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 22:38:02.052614 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1016,"mutex_wait_us":81,"thread_start_us":133,"threads_started":1}
I20260810 22:38:02.053365 4662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:02.047273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53946 (local address 127.16.233.66:44561)
0810 22:38:02.047723 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:02.047733 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:02.047978 (+ 245us) server_negotiation.cc:407] Connection header received
0810 22:38:02.048052 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:02.048057 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:02.048207 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:02.048362 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:38:02.048876 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:02.050004 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:02.051413 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:02.052281 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:02.052379 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:02.052506 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:02.052914 (+ 408us) server_negotiation.cc:299] Negotiation successful
0810 22:38:02.053017 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":144,"threads_started":1}
I20260810 22:38:02.054323 4661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:02.047099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.66:44561 (local address 127.0.0.1:53946)
0810 22:38:02.047855 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:02.047868 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 22:38:02.047965 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:38:02.048530 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:38:02.048533 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:38:02.048581 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:38:02.048697 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:02.048706 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:02.050144 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:02.050147 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:02.051300 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:02.051307 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:02.052595 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:02.052599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:02.052685 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:02.052708 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 22:38:02.052782 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 22:38:02.053132 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"mutex_wait_us":1,"thread_start_us":96,"threads_started":1}
I20260810 22:38:02.056227 4259 tablet_service.cc:1467] Processing CreateTablet for tablet 7137a893508842a49d23bd9094417c1c (DEFAULT_TABLE table=test-table [id=63db044c8004406399eed57a6e6fe473]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:38:02.056322 4568 tablet_service.cc:1467] Processing CreateTablet for tablet 7137a893508842a49d23bd9094417c1c (DEFAULT_TABLE table=test-table [id=63db044c8004406399eed57a6e6fe473]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:38:02.056929 4410 tablet_service.cc:1467] Processing CreateTablet for tablet 7137a893508842a49d23bd9094417c1c (DEFAULT_TABLE table=test-table [id=63db044c8004406399eed57a6e6fe473]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:38:02.057426 4568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7137a893508842a49d23bd9094417c1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:38:02.057432 4259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7137a893508842a49d23bd9094417c1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:38:02.057826 4410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7137a893508842a49d23bd9094417c1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:38:02.064615 4663 tablet_bootstrap.cc:492] T 7137a893508842a49d23bd9094417c1c P 732a0e621f7e473ea16149e69b7568a9: Bootstrap starting.
I20260810 22:38:02.065498 4665 tablet_bootstrap.cc:492] T 7137a893508842a49d23bd9094417c1c P 28ffb60cc303403f950a133672dc0ae6: Bootstrap starting.
I20260810 22:38:02.067180 4663 tablet_bootstrap.cc:654] T 7137a893508842a49d23bd9094417c1c P 732a0e621f7e473ea16149e69b7568a9: Neither blocks nor log segments found. Creating new log.
I20260810 22:38:02.067620 4664 tablet_bootstrap.cc:492] T 7137a893508842a49d23bd9094417c1c P 7d64a7f7068449bca9996c132073afb4: Bootstrap starting.
I20260810 22:38:02.067737 4665 tablet_bootstrap.cc:654] T 7137a893508842a49d23bd9094417c1c P 28ffb60cc303403f950a133672dc0ae6: Neither blocks nor log segments found. Creating new log.
I20260810 22:38:02.068176 4663 log.cc:824] T 7137a893508842a49d23bd9094417c1c P 732a0e621f7e473ea16149e69b7568a9: Log is configured to *not* fsync() on all Append() calls
I20260810 22:38:02.069162 4665 log.cc:824] T 7137a893508842a49d23bd9094417c1c P 28ffb60cc303403f950a133672dc0ae6: Log is configured to *not* fsync() on all Append() calls
I20260810 22:38:02.069799 4664 tablet_bootstrap.cc:654] T 7137a893508842a49d23bd9094417c1c P 7d64a7f7068449bca9996c132073afb4: Neither blocks nor log segments found. Creating new log.
I20260810 22:38:02.070729 4663 tablet_bootstrap.cc:492] T 7137a893508842a49d23bd9094417c1c P 732a0e621f7e473ea16149e69b7568a9: No bootstrap required, opened a new log
I20260810 22:38:02.070942 4663 ts_tablet_manager.cc:1397] T 7137a893508842a49d23bd9094417c1c P 732a0e621f7e473ea16149e69b7568a9: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260810 22:38:02.071051 4665 tablet_bootstrap.cc:492] T 7137a893508842a49d23bd9094417c1c P 28ffb60cc303403f950a133672dc0ae6: No bootstrap required, opened a new log
I20260810 22:38:02.071079 4664 log.cc:824] T 7137a893508842a49d23bd9094417c1c P 7d64a7f7068449bca9996c132073afb4: Log is configured to *not* fsync() on all Append() calls
I20260810 22:38:02.071265 4665 ts_tablet_manager.cc:1397] T 7137a893508842a49d23bd9094417c1c P 28ffb60cc303403f950a133672dc0ae6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 22:38:02.073196 4664 tablet_bootstrap.cc:492] T 7137a893508842a49d23bd9094417c1c P 7d64a7f7068449bca9996c132073afb4: No bootstrap required, opened a new log
I20260810 22:38:02.073398 4664 ts_tablet_manager.cc:1397] T 7137a893508842a49d23bd9094417c1c P 7d64a7f7068449bca9996c132073afb4: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 22:38:02.074515 4665 raft_consensus.cc:348] T 7137a893508842a49d23bd9094417c1c P 28ffb60cc303403f950a133672dc0ae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d64a7f7068449bca9996c132073afb4" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 35873 } } peers { permanent_uuid: "732a0e621f7e473ea16149e69b7568a9" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 38629 } } peers { permanent_uuid: "28ffb60cc303403f950a133672dc0ae6" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 44561 } }
I20260810 22:38:02.074775 4665 raft_consensus.cc:374] T 7137a893508842a49d23bd9094417c1c P 28ffb60cc303403f950a133672dc0ae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:38:02.074838 4665 raft_consensus.cc:729] T 7137a893508842a49d23bd9094417c1c P 28ffb60cc303403f950a133672dc0ae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28ffb60cc303403f950a133672dc0ae6, State: Initialized, Role: FOLLOWER
I20260810 22:38:02.075302 4665 consensus_queue.cc:260] T 7137a893508842a49d23bd9094417c1c P 28ffb60cc303403f950a133672dc0ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d64a7f7068449bca9996c132073afb4" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 35873 } } peers { permanent_uuid: "732a0e621f7e473ea16149e69b7568a9" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 38629 } } peers { permanent_uuid: "28ffb60cc303403f950a133672dc0ae6" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 44561 } }
I20260810 22:38:02.076305 4665 ts_tablet_manager.cc:1428] T 7137a893508842a49d23bd9094417c1c P 28ffb60cc303403f950a133672dc0ae6: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 22:38:02.076701 4491 heartbeater.cc:503] Master 127.16.233.126:37427 was elected leader, sending a full tablet report...
I20260810 22:38:02.076898 4663 raft_consensus.cc:348] T 7137a893508842a49d23bd9094417c1c P 732a0e621f7e473ea16149e69b7568a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d64a7f7068449bca9996c132073afb4" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 35873 } } peers { permanent_uuid: "732a0e621f7e473ea16149e69b7568a9" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 38629 } } peers { permanent_uuid: "28ffb60cc303403f950a133672dc0ae6" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 44561 } }
I20260810 22:38:02.077219 4663 raft_consensus.cc:374] T 7137a893508842a49d23bd9094417c1c P 732a0e621f7e473ea16149e69b7568a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:38:02.077281 4663 raft_consensus.cc:729] T 7137a893508842a49d23bd9094417c1c P 732a0e621f7e473ea16149e69b7568a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732a0e621f7e473ea16149e69b7568a9, State: Initialized, Role: FOLLOWER
I20260810 22:38:02.077327 4664 raft_consensus.cc:348] T 7137a893508842a49d23bd9094417c1c P 7d64a7f7068449bca9996c132073afb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d64a7f7068449bca9996c132073afb4" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 35873 } } peers { permanent_uuid: "732a0e621f7e473ea16149e69b7568a9" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 38629 } } peers { permanent_uuid: "28ffb60cc303403f950a133672dc0ae6" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 44561 } }
I20260810 22:38:02.077571 4664 raft_consensus.cc:374] T 7137a893508842a49d23bd9094417c1c P 7d64a7f7068449bca9996c132073afb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:38:02.077631 4664 raft_consensus.cc:729] T 7137a893508842a49d23bd9094417c1c P 7d64a7f7068449bca9996c132073afb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d64a7f7068449bca9996c132073afb4, State: Initialized, Role: FOLLOWER
I20260810 22:38:02.077731 4663 consensus_queue.cc:260] T 7137a893508842a49d23bd9094417c1c P 732a0e621f7e473ea16149e69b7568a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d64a7f7068449bca9996c132073afb4" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 35873 } } peers { permanent_uuid: "732a0e621f7e473ea16149e69b7568a9" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 38629 } } peers { permanent_uuid: "28ffb60cc303403f950a133672dc0ae6" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 44561 } }
I20260810 22:38:02.078079 4664 consensus_queue.cc:260] T 7137a893508842a49d23bd9094417c1c P 7d64a7f7068449bca9996c132073afb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d64a7f7068449bca9996c132073afb4" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 35873 } } peers { permanent_uuid: "732a0e621f7e473ea16149e69b7568a9" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 38629 } } peers { permanent_uuid: "28ffb60cc303403f950a133672dc0ae6" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 44561 } }
I20260810 22:38:02.078548 4333 heartbeater.cc:503] Master 127.16.233.126:37427 was elected leader, sending a full tablet report...
I20260810 22:38:02.078796 4663 ts_tablet_manager.cc:1428] T 7137a893508842a49d23bd9094417c1c P 732a0e621f7e473ea16149e69b7568a9: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260810 22:38:02.078965 4664 ts_tablet_manager.cc:1428] T 7137a893508842a49d23bd9094417c1c P 7d64a7f7068449bca9996c132073afb4: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 22:38:02.079296 4643 heartbeater.cc:503] Master 127.16.233.126:37427 was elected leader, sending a full tablet report...
W20260810 22:38:02.240731 4644 tablet.cc:2369] T 7137a893508842a49d23bd9094417c1c P 7d64a7f7068449bca9996c132073afb4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 22:38:02.249953 4492 tablet.cc:2369] T 7137a893508842a49d23bd9094417c1c P 28ffb60cc303403f950a133672dc0ae6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 22:38:02.262486 4334 tablet.cc:2369] T 7137a893508842a49d23bd9094417c1c P 732a0e621f7e473ea16149e69b7568a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:38:02.362517 4669 raft_consensus.cc:482] T 7137a893508842a49d23bd9094417c1c P 28ffb60cc303403f950a133672dc0ae6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:38:02.362846 4669 raft_consensus.cc:504] T 7137a893508842a49d23bd9094417c1c P 28ffb60cc303403f950a133672dc0ae6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d64a7f7068449bca9996c132073afb4" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 35873 } } peers { permanent_uuid: "732a0e621f7e473ea16149e69b7568a9" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 38629 } } peers { permanent_uuid: "28ffb60cc303403f950a133672dc0ae6" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 44561 } }
I20260810 22:38:02.364243 4669 leader_election.cc:288] T 7137a893508842a49d23bd9094417c1c P 28ffb60cc303403f950a133672dc0ae6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d64a7f7068449bca9996c132073afb4 (127.16.233.67:35873), 732a0e621f7e473ea16149e69b7568a9 (127.16.233.65:38629)
I20260810 22:38:02.368772 4660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:02.364722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:36253 (local address 127.16.233.65:38629)
0810 22:38:02.364992 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:02.364996 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:02.365105 (+ 109us) server_negotiation.cc:407] Connection header received
0810 22:38:02.365264 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:02.365267 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:02.365316 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:02.365398 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:38:02.365896 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:02.366462 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:02.367713 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:02.368141 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:02.368279 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:02.368337 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:02.368555 (+ 218us) server_negotiation.cc:299] Negotiation successful
0810 22:38:02.368634 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260810 22:38:02.368784 4672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:02.364551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.67:35873 (local address 127.16.233.66:35021)
0810 22:38:02.365151 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:02.365170 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 22:38:02.365295 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:38:02.365641 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:38:02.365643 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:38:02.365659 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:38:02.365842 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:02.365850 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:02.366716 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:02.366719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:02.367583 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:02.367592 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:02.368306 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:02.368309 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:02.368442 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:02.368460 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 22:38:02.368555 (+ 95us) client_negotiation.cc:240] Negotiation successful
0810 22:38:02.368629 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"mutex_wait_us":85,"thread_start_us":85,"threads_started":1}
I20260810 22:38:02.368778 4659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:02.364709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:35021 (local address 127.16.233.67:35873)
0810 22:38:02.364897 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:02.364903 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:02.365299 (+ 396us) server_negotiation.cc:407] Connection header received
0810 22:38:02.365443 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:02.365447 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:02.365508 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:02.365616 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:38:02.366001 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:02.366592 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:02.367751 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:02.368150 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:02.368292 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:02.368360 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:02.368592 (+ 232us) server_negotiation.cc:299] Negotiation successful
0810 22:38:02.368672 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 22:38:02.368793 4673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:02.364551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.65:38629 (local address 127.16.233.66:36253)
0810 22:38:02.364977 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:02.364995 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 22:38:02.365135 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:38:02.365440 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:38:02.365444 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:38:02.365467 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:38:02.365737 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:02.365750 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:02.366604 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:02.366608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:02.367572 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:02.367582 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:02.368279 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:02.368282 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:02.368387 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:02.368445 (+ 58us) client_negotiation.cc:769] Sending connection context
0810 22:38:02.368508 (+ 63us) client_negotiation.cc:240] Negotiation successful
0810 22:38:02.368613 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":70,"thread_start_us":86,"threads_started":1}
I20260810 22:38:02.369652 4590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7137a893508842a49d23bd9094417c1c" candidate_uuid: "28ffb60cc303403f950a133672dc0ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d64a7f7068449bca9996c132073afb4" is_pre_election: true
I20260810 22:38:02.370034 4590 raft_consensus.cc:2454] T 7137a893508842a49d23bd9094417c1c P 7d64a7f7068449bca9996c132073afb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28ffb60cc303403f950a133672dc0ae6 in term 0.
I20260810 22:38:02.370008 4283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7137a893508842a49d23bd9094417c1c" candidate_uuid: "28ffb60cc303403f950a133672dc0ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732a0e621f7e473ea16149e69b7568a9" is_pre_election: true
I20260810 22:38:02.370285 4283 raft_consensus.cc:2454] T 7137a893508842a49d23bd9094417c1c P 732a0e621f7e473ea16149e69b7568a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28ffb60cc303403f950a133672dc0ae6 in term 0.
I20260810 22:38:02.370607 4356 leader_election.cc:302] T 7137a893508842a49d23bd9094417c1c P 28ffb60cc303403f950a133672dc0ae6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28ffb60cc303403f950a133672dc0ae6, 7d64a7f7068449bca9996c132073afb4; no voters:
I20260810 22:38:02.370971 4669 raft_consensus.cc:2781] T 7137a893508842a49d23bd9094417c1c P 28ffb60cc303403f950a133672dc0ae6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:38:02.371078 4669 raft_consensus.cc:482] T 7137a893508842a49d23bd9094417c1c P 28ffb60cc303403f950a133672dc0ae6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:38:02.371140 4669 raft_consensus.cc:3037] T 7137a893508842a49d23bd9094417c1c P 28ffb60cc303403f950a133672dc0ae6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:38:02.372272 4669 raft_consensus.cc:504] T 7137a893508842a49d23bd9094417c1c P 28ffb60cc303403f950a133672dc0ae6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d64a7f7068449bca9996c132073afb4" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 35873 } } peers { permanent_uuid: "732a0e621f7e473ea16149e69b7568a9" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 38629 } } peers { permanent_uuid: "28ffb60cc303403f950a133672dc0ae6" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 44561 } }
I20260810 22:38:02.372670 4669 leader_election.cc:288] T 7137a893508842a49d23bd9094417c1c P 28ffb60cc303403f950a133672dc0ae6 [CANDIDATE]: Term 1 election: Requested vote from peers 7d64a7f7068449bca9996c132073afb4 (127.16.233.67:35873), 732a0e621f7e473ea16149e69b7568a9 (127.16.233.65:38629)
I20260810 22:38:02.373085 4283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7137a893508842a49d23bd9094417c1c" candidate_uuid: "28ffb60cc303403f950a133672dc0ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "732a0e621f7e473ea16149e69b7568a9"
I20260810 22:38:02.373154 4590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7137a893508842a49d23bd9094417c1c" candidate_uuid: "28ffb60cc303403f950a133672dc0ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d64a7f7068449bca9996c132073afb4"
I20260810 22:38:02.373261 4283 raft_consensus.cc:3037] T 7137a893508842a49d23bd9094417c1c P 732a0e621f7e473ea16149e69b7568a9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:38:02.373288 4590 raft_consensus.cc:3037] T 7137a893508842a49d23bd9094417c1c P 7d64a7f7068449bca9996c132073afb4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:38:02.374394 4283 raft_consensus.cc:2454] T 7137a893508842a49d23bd9094417c1c P 732a0e621f7e473ea16149e69b7568a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28ffb60cc303403f950a133672dc0ae6 in term 1.
I20260810 22:38:02.374402 4590 raft_consensus.cc:2454] T 7137a893508842a49d23bd9094417c1c P 7d64a7f7068449bca9996c132073afb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28ffb60cc303403f950a133672dc0ae6 in term 1.
I20260810 22:38:02.374759 4356 leader_election.cc:302] T 7137a893508842a49d23bd9094417c1c P 28ffb60cc303403f950a133672dc0ae6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28ffb60cc303403f950a133672dc0ae6, 7d64a7f7068449bca9996c132073afb4; no voters:
I20260810 22:38:02.374965 4669 raft_consensus.cc:2781] T 7137a893508842a49d23bd9094417c1c P 28ffb60cc303403f950a133672dc0ae6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:38:02.375236 4669 raft_consensus.cc:686] T 7137a893508842a49d23bd9094417c1c P 28ffb60cc303403f950a133672dc0ae6 [term 1 LEADER]: Becoming Leader. State: Replica: 28ffb60cc303403f950a133672dc0ae6, State: Running, Role: LEADER
I20260810 22:38:02.375655 4669 consensus_queue.cc:237] T 7137a893508842a49d23bd9094417c1c P 28ffb60cc303403f950a133672dc0ae6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d64a7f7068449bca9996c132073afb4" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 35873 } } peers { permanent_uuid: "732a0e621f7e473ea16149e69b7568a9" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 38629 } } peers { permanent_uuid: "28ffb60cc303403f950a133672dc0ae6" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 44561 } }
I20260810 22:38:02.379211 4123 catalog_manager.cc:5495] T 7137a893508842a49d23bd9094417c1c P 28ffb60cc303403f950a133672dc0ae6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28ffb60cc303403f950a133672dc0ae6 (127.16.233.66). New cstate: current_term: 1 leader_uuid: "28ffb60cc303403f950a133672dc0ae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d64a7f7068449bca9996c132073afb4" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 35873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "732a0e621f7e473ea16149e69b7568a9" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 38629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28ffb60cc303403f950a133672dc0ae6" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 44561 } health_report { overall_health: HEALTHY } } }
I20260810 22:38:02.422727 4662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:02.417722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53950 (local address 127.16.233.66:44561)
0810 22:38:02.417903 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:02.417911 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:02.417930 (+ 19us) server_negotiation.cc:407] Connection header received
0810 22:38:02.418164 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:02.418167 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:02.418235 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:02.418378 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:38:02.418681 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:02.419312 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:02.420406 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:02.420929 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:02.421048 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:02.421566 (+ 518us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:38:02.422247 (+ 681us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:38:02.422365 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:02.422463 (+ 98us) server_negotiation.cc:299] Negotiation successful
0810 22:38:02.422527 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 22:38:02.429948 4283 raft_consensus.cc:1261] T 7137a893508842a49d23bd9094417c1c P 732a0e621f7e473ea16149e69b7568a9 [term 1 FOLLOWER]: Refusing update from remote peer 28ffb60cc303403f950a133672dc0ae6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:38:02.429947 4590 raft_consensus.cc:1261] T 7137a893508842a49d23bd9094417c1c P 7d64a7f7068449bca9996c132073afb4 [term 1 FOLLOWER]: Refusing update from remote peer 28ffb60cc303403f950a133672dc0ae6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:38:02.430754 4674 consensus_queue.cc:1035] T 7137a893508842a49d23bd9094417c1c P 28ffb60cc303403f950a133672dc0ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "732a0e621f7e473ea16149e69b7568a9" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 38629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:38:02.430963 4669 consensus_queue.cc:1035] T 7137a893508842a49d23bd9094417c1c P 28ffb60cc303403f950a133672dc0ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d64a7f7068449bca9996c132073afb4" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 35873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:38:02.442127 4679 mvcc.cc:205] Tried to move back new op lower bound from 7317100472023879680 to 7317100471813488640. Current Snapshot: MvccSnapshot[applied={T|T < 7317100472023879680}]
I20260810 22:38:02.448318 4123 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:38:02.448516 4123 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:38:02.451558 4123 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5ae784d29260410b8c807b3121b7671e: Sending DeleteTablet for 3 replicas of tablet 7137a893508842a49d23bd9094417c1c
I20260810 22:38:02.452621 4259 tablet_service.cc:1514] Processing DeleteTablet for tablet 7137a893508842a49d23bd9094417c1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:38:02 UTC) from {username='kudu', principal='kudu/127.16.233.126@KRBTEST.COM'} at 127.0.0.1:40384
I20260810 22:38:02.452636 4568 tablet_service.cc:1514] Processing DeleteTablet for tablet 7137a893508842a49d23bd9094417c1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:38:02 UTC) from {username='kudu', principal='kudu/127.16.233.126@KRBTEST.COM'} at 127.0.0.1:49832
I20260810 22:38:02.452775 4410 tablet_service.cc:1514] Processing DeleteTablet for tablet 7137a893508842a49d23bd9094417c1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:38:02 UTC) from {username='kudu', principal='kudu/127.16.233.126@KRBTEST.COM'} at 127.0.0.1:53946
I20260810 22:38:02.453260 4690 tablet_replica.cc:330] T 7137a893508842a49d23bd9094417c1c P 28ffb60cc303403f950a133672dc0ae6: stopping tablet replica
I20260810 22:38:02.453562 4688 tablet_replica.cc:330] T 7137a893508842a49d23bd9094417c1c P 7d64a7f7068449bca9996c132073afb4: stopping tablet replica
I20260810 22:38:02.453641 4690 raft_consensus.cc:2229] T 7137a893508842a49d23bd9094417c1c P 28ffb60cc303403f950a133672dc0ae6 [term 1 LEADER]: Raft consensus shutting down.
I20260810 22:38:02.453749 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 4191
I20260810 22:38:02.454018 4688 raft_consensus.cc:2229] T 7137a893508842a49d23bd9094417c1c P 7d64a7f7068449bca9996c132073afb4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:38:02.454416 4688 raft_consensus.cc:2258] T 7137a893508842a49d23bd9094417c1c P 7d64a7f7068449bca9996c132073afb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:38:02.456638 4688 ts_tablet_manager.cc:1905] T 7137a893508842a49d23bd9094417c1c P 7d64a7f7068449bca9996c132073afb4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:38:02.460180 4688 ts_tablet_manager.cc:1918] T 7137a893508842a49d23bd9094417c1c P 7d64a7f7068449bca9996c132073afb4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:38:02.460372 4688 log.cc:1196] T 7137a893508842a49d23bd9094417c1c P 7d64a7f7068449bca9996c132073afb4: Deleting WAL directory at /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/minicluster-data/ts-2/wal/wals/7137a893508842a49d23bd9094417c1c
I20260810 22:38:02.460779 4688 ts_tablet_manager.cc:1939] T 7137a893508842a49d23bd9094417c1c P 7d64a7f7068449bca9996c132073afb4: Deleting consensus metadata
I20260810 22:38:02.462685 4690 raft_consensus.cc:2258] T 7137a893508842a49d23bd9094417c1c P 28ffb60cc303403f950a133672dc0ae6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:38:02.462723 4098 catalog_manager.cc:4841] TS 7d64a7f7068449bca9996c132073afb4 (127.16.233.67:35873): tablet 7137a893508842a49d23bd9094417c1c (table test-table [id=63db044c8004406399eed57a6e6fe473]) successfully deleted
I20260810 22:38:02.465050 4690 ts_tablet_manager.cc:1905] T 7137a893508842a49d23bd9094417c1c P 28ffb60cc303403f950a133672dc0ae6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:38:02.469044 4690 ts_tablet_manager.cc:1918] T 7137a893508842a49d23bd9094417c1c P 28ffb60cc303403f950a133672dc0ae6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:38:02.469205 4690 log.cc:1196] T 7137a893508842a49d23bd9094417c1c P 28ffb60cc303403f950a133672dc0ae6: Deleting WAL directory at /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786401215402979-17317-0/minicluster-data/ts-1/wal/wals/7137a893508842a49d23bd9094417c1c
I20260810 22:38:02.469578 4690 ts_tablet_manager.cc:1939] T 7137a893508842a49d23bd9094417c1c P 28ffb60cc303403f950a133672dc0ae6: Deleting consensus metadata
I20260810 22:38:02.471112 4098 catalog_manager.cc:4841] TS 28ffb60cc303403f950a133672dc0ae6 (127.16.233.66:44561): tablet 7137a893508842a49d23bd9094417c1c (table test-table [id=63db044c8004406399eed57a6e6fe473]) successfully deleted
W20260810 22:38:02.471407 4100 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.233.65:38629 (error 108)
I20260810 22:38:02.472476 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 4338
I20260810 22:38:02.472710 4661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:02.472330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.65:38629 (local address 127.0.0.1:40390)
0810 22:38:02.472509 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:02.472598 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.65:38629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260810 22:38:02.474149 4100 catalog_manager.cc:4584] TS 732a0e621f7e473ea16149e69b7568a9 (127.16.233.65:38629): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7137a893508842a49d23bd9094417c1c: Network error: Client connection negotiation failed: client connection to 127.16.233.65:38629: connect: Connection refused (error 111)
I20260810 22:38:02.481165 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 4497
I20260810 22:38:02.488767 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 4080
2026-08-10T22:38:02Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2802 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-10T22:38:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:38:02Z Disabled control of system clock
I20260810 22:38:02.525820 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786401215402979-17317-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786401215402979-17317-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786401215402979-17317-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:44791
--webserver_interface=127.16.233.126
--webserver_port=0
--builtin_ntp_servers=127.16.233.84:36745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.126:44791
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 22:38:02.645375 4698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:38:02.645675 4698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:38:02.645713 4698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:38:02.649974 4698 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:38:02.650050 4698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:38:02.650065 4698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:38:02.650084 4698 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:38:02.650137 4698 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:38:02.655516 4698 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:36745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786401215402979-17317-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786401215402979-17317-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.126:44791
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:44791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786401215402979-17317-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:38:02.657244 4698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:38:02.659354 4698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:38:02.671180 4705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:38:02.671253 4698 server_base.cc:1034] running on GCE node
W20260810 22:38:02.671145 4704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:02.671200 4707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:38:02.672091 4698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:38:02.673478 4698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:38:02.674678 4698 hybrid_clock.cc:648] HybridClock initialized: now 1786401482674641 us; error 53 us; skew 500 ppm
I20260810 22:38:02.676967 4698 webserver.cc:460] Webserver started at http://127.16.233.126:32877/ using document root <none> and password file <none>
I20260810 22:38:02.677654 4698 fs_manager.cc:359] Metadata directory not provided
I20260810 22:38:02.677728 4698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:38:02.678040 4698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:38:02.680531 4698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786401215402979-17317-0/minicluster-data/master-0/data/instance:
uuid: "6aad93b30525408ea2810a23d883461a"
format_stamp: "Formatted at 2026-08-10 22:38:02 on dist-test-slave-77v9"
server_key: "073e885d1b055e62857278a0b67d1ba5"
server_key_iv: "5bf823c209ddd9a398ef5892a5cef622"
server_key_version: "encryptionkey@0"
I20260810 22:38:02.681327 4698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786401215402979-17317-0/minicluster-data/master-0/wal/instance:
uuid: "6aad93b30525408ea2810a23d883461a"
format_stamp: "Formatted at 2026-08-10 22:38:02 on dist-test-slave-77v9"
server_key: "073e885d1b055e62857278a0b67d1ba5"
server_key_iv: "5bf823c209ddd9a398ef5892a5cef622"
server_key_version: "encryptionkey@0"
I20260810 22:38:02.685796 4698 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 22:38:02.688772 4715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:02.690325 4698 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 22:38:02.690496 4698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786401215402979-17317-0/minicluster-data/master-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786401215402979-17317-0/minicluster-data/master-0/wal
uuid: "6aad93b30525408ea2810a23d883461a"
format_stamp: "Formatted at 2026-08-10 22:38:02 on dist-test-slave-77v9"
server_key: "073e885d1b055e62857278a0b67d1ba5"
server_key_iv: "5bf823c209ddd9a398ef5892a5cef622"
server_key_version: "encryptionkey@0"
I20260810 22:38:02.690634 4698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786401215402979-17317-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786401215402979-17317-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786401215402979-17317-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:38:02.710798 4698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:38:02.711674 4698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:38:02.711927 4698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:38:02.720391 4773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.126:44791 every 8 connection(s)
I20260810 22:38:02.720388 4698 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.126:44791
I20260810 22:38:02.721752 4698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
I20260810 22:38:02.723510 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 4698
I20260810 22:38:02.723695 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786401215402979-17317-0/minicluster-data/master-0/wal/instance
I20260810 22:38:02.724021 17317 external_mini_cluster.cc:1428] Setting key 2d14a277312f7448af58528a9c57318f
I20260810 22:38:02.725831 4774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:38:02.732535 4774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6aad93b30525408ea2810a23d883461a: Bootstrap starting.
I20260810 22:38:02.734997 4778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:02.725736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38556 (local address 127.16.233.126:44791)
0810 22:38:02.726503 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:02.726521 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:02.726562 (+ 41us) server_negotiation.cc:407] Connection header received
0810 22:38:02.727295 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:02.727313 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:02.727744 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:02.728195 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:02.728708 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:02.729908 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:02.730925 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:02.731467 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:02.731701 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:02.732773 (+ 1072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:02.732780 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:02.732797 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:38:02.733286 (+ 489us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:02.733347 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:38:02.733378 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:02.733504 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:02.733675 (+ 171us) server_negotiation.cc:299] Negotiation successful
0810 22:38:02.733983 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":172,"threads_started":1}
I20260810 22:38:02.735643 4774 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6aad93b30525408ea2810a23d883461a: Neither blocks nor log segments found. Creating new log.
I20260810 22:38:02.736689 4774 log.cc:824] T 00000000000000000000000000000000 P 6aad93b30525408ea2810a23d883461a: Log is configured to *not* fsync() on all Append() calls
I20260810 22:38:02.739135 4774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6aad93b30525408ea2810a23d883461a: No bootstrap required, opened a new log
I20260810 22:38:02.743268 4774 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6aad93b30525408ea2810a23d883461a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aad93b30525408ea2810a23d883461a" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 44791 } }
I20260810 22:38:02.743557 4774 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6aad93b30525408ea2810a23d883461a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:38:02.743633 4774 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6aad93b30525408ea2810a23d883461a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6aad93b30525408ea2810a23d883461a, State: Initialized, Role: FOLLOWER
I20260810 22:38:02.744189 4774 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6aad93b30525408ea2810a23d883461a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aad93b30525408ea2810a23d883461a" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 44791 } }
I20260810 22:38:02.744374 4774 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6aad93b30525408ea2810a23d883461a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:38:02.744467 4774 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6aad93b30525408ea2810a23d883461a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:38:02.744596 4774 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6aad93b30525408ea2810a23d883461a [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:38:02.745818 4774 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6aad93b30525408ea2810a23d883461a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aad93b30525408ea2810a23d883461a" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 44791 } }
I20260810 22:38:02.746243 4774 leader_election.cc:302] T 00000000000000000000000000000000 P 6aad93b30525408ea2810a23d883461a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6aad93b30525408ea2810a23d883461a; no voters:
I20260810 22:38:02.746670 4774 leader_election.cc:288] T 00000000000000000000000000000000 P 6aad93b30525408ea2810a23d883461a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:38:02.746901 4780 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6aad93b30525408ea2810a23d883461a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:38:02.747213 4780 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6aad93b30525408ea2810a23d883461a [term 1 LEADER]: Becoming Leader. State: Replica: 6aad93b30525408ea2810a23d883461a, State: Running, Role: LEADER
I20260810 22:38:02.747602 4780 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6aad93b30525408ea2810a23d883461a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aad93b30525408ea2810a23d883461a" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 44791 } }
I20260810 22:38:02.748230 4774 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6aad93b30525408ea2810a23d883461a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:38:02.749749 4782 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6aad93b30525408ea2810a23d883461a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6aad93b30525408ea2810a23d883461a. Latest consensus state: current_term: 1 leader_uuid: "6aad93b30525408ea2810a23d883461a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aad93b30525408ea2810a23d883461a" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 44791 } } }
I20260810 22:38:02.749749 4781 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6aad93b30525408ea2810a23d883461a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6aad93b30525408ea2810a23d883461a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aad93b30525408ea2810a23d883461a" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 44791 } } }
I20260810 22:38:02.749933 4781 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6aad93b30525408ea2810a23d883461a [sys.catalog]: This master's current role is: LEADER
I20260810 22:38:02.750213 4782 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6aad93b30525408ea2810a23d883461a [sys.catalog]: This master's current role is: LEADER
I20260810 22:38:02.750392 4789 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:38:02.754114 4789 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:38:02.761198 4789 catalog_manager.cc:1347] Generated new cluster ID: b11af9dcf16947e985d25069a26c56c1
I20260810 22:38:02.761309 4789 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:38:02.783699 4789 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:38:02.784919 4789 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:38:02.791391 4789 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6aad93b30525408ea2810a23d883461a: Generated new TSK 0
I20260810 22:38:02.792291 4789 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:38:02.803442 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786401215402979-17317-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786401215402979-17317-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786401215402979-17317-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.65:0
--local_ip_for_outbound_sockets=127.16.233.65
--webserver_interface=127.16.233.65
--webserver_port=0
--tserver_master_addrs=127.16.233.126:44791
--builtin_ntp_servers=127.16.233.84:36745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 22:38:02.945864 4800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:38:02.946177 4800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:38:02.946218 4800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:38:02.950587 4800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:38:02.950660 4800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:38:02.950758 4800 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.65
I20260810 22:38:02.955862 4800 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:36745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786401215402979-17317-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786401215402979-17317-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.233.65
--webserver_port=0
--tserver_master_addrs=127.16.233.126:44791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.65
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786401215402979-17317-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:38:02.957095 4800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:38:02.958598 4800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 22:38:02.968871 4800 server_base.cc:1034] running on GCE node
W20260810 22:38:02.968780 4806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:02.968768 4805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:02.968768 4808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:38:02.969525 4800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:38:02.970213 4800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:38:02.971397 4800 hybrid_clock.cc:648] HybridClock initialized: now 1786401482971361 us; error 46 us; skew 500 ppm
I20260810 22:38:02.973567 4800 webserver.cc:460] Webserver started at http://127.16.233.65:37335/ using document root <none> and password file <none>
I20260810 22:38:02.974192 4800 fs_manager.cc:359] Metadata directory not provided
I20260810 22:38:02.974259 4800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:38:02.974485 4800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:38:02.976717 4800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786401215402979-17317-0/minicluster-data/ts-0/data/instance:
uuid: "d2c6ac572bb54b4192a32044af5ed996"
format_stamp: "Formatted at 2026-08-10 22:38:02 on dist-test-slave-77v9"
server_key: "3f31836e87949fab37844f432df5416a"
server_key_iv: "8ce78f340e70157b64def0ae8a15cb17"
server_key_version: "encryptionkey@0"
I20260810 22:38:02.977397 4800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786401215402979-17317-0/minicluster-data/ts-0/wal/instance:
uuid: "d2c6ac572bb54b4192a32044af5ed996"
format_stamp: "Formatted at 2026-08-10 22:38:02 on dist-test-slave-77v9"
server_key: "3f31836e87949fab37844f432df5416a"
server_key_iv: "8ce78f340e70157b64def0ae8a15cb17"
server_key_version: "encryptionkey@0"
I20260810 22:38:02.981616 4800 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 22:38:02.984338 4816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:02.985651 4800 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 22:38:02.985797 4800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786401215402979-17317-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786401215402979-17317-0/minicluster-data/ts-0/wal
uuid: "d2c6ac572bb54b4192a32044af5ed996"
format_stamp: "Formatted at 2026-08-10 22:38:02 on dist-test-slave-77v9"
server_key: "3f31836e87949fab37844f432df5416a"
server_key_iv: "8ce78f340e70157b64def0ae8a15cb17"
server_key_version: "encryptionkey@0"
I20260810 22:38:02.985946 4800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786401215402979-17317-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786401215402979-17317-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786401215402979-17317-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:38:02.997748 4800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:38:02.998500 4800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:38:02.998719 4800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:38:02.999378 4800 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:38:03.000495 4800 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:38:03.000563 4800 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:03.000653 4800 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:38:03.000698 4800 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:03.009878 4800 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.65:34753
I20260810 22:38:03.009984 4945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.65:34753 every 8 connection(s)
I20260810 22:38:03.010864 4800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
I20260810 22:38:03.013222 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 4800
I20260810 22:38:03.013355 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786401215402979-17317-0/minicluster-data/ts-0/wal/instance
I20260810 22:38:03.013669 17317 external_mini_cluster.cc:1428] Setting key 151ba944adbeb5811dae656907df6b40
I20260810 22:38:03.015978 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786401215402979-17317-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786401215402979-17317-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786401215402979-17317-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.66:0
--local_ip_for_outbound_sockets=127.16.233.66
--webserver_interface=127.16.233.66
--webserver_port=0
--tserver_master_addrs=127.16.233.126:44791
--builtin_ntp_servers=127.16.233.84:36745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 22:38:03.022213 4778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:03.012876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:57449 (local address 127.16.233.126:44791)
0810 22:38:03.013064 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:03.013069 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:03.014355 (+ 1286us) server_negotiation.cc:407] Connection header received
0810 22:38:03.016198 (+ 1843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:03.016204 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:03.016274 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:03.016344 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:03.017949 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:03.018847 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:03.020059 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:03.020287 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:03.020408 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:03.021656 (+ 1248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:03.021660 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:03.021663 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:38:03.021682 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:03.021702 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:38:03.021710 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:03.021790 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:03.021961 (+ 171us) server_negotiation.cc:299] Negotiation successful
0810 22:38:03.022056 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260810 22:38:03.023141 4949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:03.013361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:44791 (local address 127.16.233.65:57449)
0810 22:38:03.014162 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:03.014207 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 22:38:03.015929 (+ 1722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:38:03.016783 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:38:03.016795 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:38:03.016986 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:38:03.017737 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:03.017760 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:03.019018 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:03.019021 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:03.019924 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:03.019934 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:03.020460 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:03.020465 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:03.020640 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:03.021365 (+ 725us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 22:38:03.021367 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:38:03.021408 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 22:38:03.021475 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:38:03.021792 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:38:03.021804 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:38:03.021810 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 22:38:03.021947 (+ 137us) client_negotiation.cc:240] Negotiation successful
0810 22:38:03.022250 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":128,"threads_started":1}
I20260810 22:38:03.024567 4947 heartbeater.cc:347] Connected to a master server at 127.16.233.126:44791
I20260810 22:38:03.024886 4947 heartbeater.cc:464] Registering TS with master...
I20260810 22:38:03.025487 4947 heartbeater.cc:511] Master 127.16.233.126:44791 requested a full tablet report, sending...
I20260810 22:38:03.027417 4735 ts_manager.cc:194] Registered new tserver with Master: d2c6ac572bb54b4192a32044af5ed996 (127.16.233.65:34753)
I20260810 22:38:03.029359 4735 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.233.65:57449
W20260810 22:38:03.137298 4950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:38:03.137599 4950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:38:03.137642 4950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:38:03.141705 4950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:38:03.141768 4950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:38:03.141832 4950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.66
I20260810 22:38:03.146974 4950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:36745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786401215402979-17317-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786401215402979-17317-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.233.66
--webserver_port=0
--tserver_master_addrs=127.16.233.126:44791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4950
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.66
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786401215402979-17317-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:38:03.148214 4950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:38:03.149749 4950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:38:03.160105 4959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:03.160109 4957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:38:03.160348 4950 server_base.cc:1034] running on GCE node
W20260810 22:38:03.160105 4956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:38:03.160995 4950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:38:03.161742 4950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:38:03.162925 4950 hybrid_clock.cc:648] HybridClock initialized: now 1786401483162904 us; error 34 us; skew 500 ppm
I20260810 22:38:03.165185 4950 webserver.cc:460] Webserver started at http://127.16.233.66:38135/ using document root <none> and password file <none>
I20260810 22:38:03.165834 4950 fs_manager.cc:359] Metadata directory not provided
I20260810 22:38:03.165903 4950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:38:03.166153 4950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:38:03.168377 4950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786401215402979-17317-0/minicluster-data/ts-1/data/instance:
uuid: "cb8a097762ea4a968eaedf8d360cf1cb"
format_stamp: "Formatted at 2026-08-10 22:38:03 on dist-test-slave-77v9"
server_key: "85a232788f3d377c1139b578c39fcbfa"
server_key_iv: "258f22de8cb23695f73c3d9b8e77466d"
server_key_version: "encryptionkey@0"
I20260810 22:38:03.169026 4950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786401215402979-17317-0/minicluster-data/ts-1/wal/instance:
uuid: "cb8a097762ea4a968eaedf8d360cf1cb"
format_stamp: "Formatted at 2026-08-10 22:38:03 on dist-test-slave-77v9"
server_key: "85a232788f3d377c1139b578c39fcbfa"
server_key_iv: "258f22de8cb23695f73c3d9b8e77466d"
server_key_version: "encryptionkey@0"
I20260810 22:38:03.173143 4950 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 22:38:03.175827 4965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:03.177266 4950 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:38:03.177407 4950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786401215402979-17317-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786401215402979-17317-0/minicluster-data/ts-1/wal
uuid: "cb8a097762ea4a968eaedf8d360cf1cb"
format_stamp: "Formatted at 2026-08-10 22:38:03 on dist-test-slave-77v9"
server_key: "85a232788f3d377c1139b578c39fcbfa"
server_key_iv: "258f22de8cb23695f73c3d9b8e77466d"
server_key_version: "encryptionkey@0"
I20260810 22:38:03.177532 4950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786401215402979-17317-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786401215402979-17317-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786401215402979-17317-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:38:03.189788 4950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:38:03.190590 4950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:38:03.190816 4950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:38:03.191459 4950 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:38:03.192459 4950 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:38:03.192509 4950 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:03.192584 4950 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:38:03.192620 4950 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:03.202221 4950 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.66:37561
I20260810 22:38:03.202280 5095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.66:37561 every 8 connection(s)
I20260810 22:38:03.203215 4950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
I20260810 22:38:03.204735 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 4950
I20260810 22:38:03.204970 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786401215402979-17317-0/minicluster-data/ts-1/wal/instance
I20260810 22:38:03.205335 17317 external_mini_cluster.cc:1428] Setting key af881852a5171d563b139f52e9b5e1d0
I20260810 22:38:03.207553 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786401215402979-17317-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786401215402979-17317-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786401215402979-17317-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.67:0
--local_ip_for_outbound_sockets=127.16.233.67
--webserver_interface=127.16.233.67
--webserver_port=0
--tserver_master_addrs=127.16.233.126:44791
--builtin_ntp_servers=127.16.233.84:36745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 22:38:03.213799 4778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:03.205374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:47839 (local address 127.16.233.126:44791)
0810 22:38:03.205544 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:03.205548 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:03.206903 (+ 1355us) server_negotiation.cc:407] Connection header received
0810 22:38:03.208533 (+ 1630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:03.208539 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:03.208608 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:03.208710 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:03.209772 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:03.210309 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:03.211515 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:03.211755 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:03.211881 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:03.213260 (+ 1379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:03.213264 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:03.213267 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:38:03.213285 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:03.213312 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:38:03.213320 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:03.213407 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:03.213571 (+ 164us) server_negotiation.cc:299] Negotiation successful
0810 22:38:03.213643 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 22:38:03.214906 5098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:03.205935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:44791 (local address 127.16.233.66:47839)
0810 22:38:03.206717 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:03.206805 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0810 22:38:03.208278 (+ 1473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:38:03.209001 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:38:03.209008 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:38:03.209098 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:38:03.209608 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:03.209621 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:03.210459 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:03.210463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:03.211374 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:03.211385 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:03.211929 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:03.211933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:03.212112 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:03.212969 (+ 857us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 22:38:03.212970 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:38:03.213018 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 22:38:03.213088 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:38:03.213408 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:38:03.213420 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:38:03.213429 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 22:38:03.213582 (+ 153us) client_negotiation.cc:240] Negotiation successful
0810 22:38:03.213902 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":132,"threads_started":1}
I20260810 22:38:03.216231 5096 heartbeater.cc:347] Connected to a master server at 127.16.233.126:44791
I20260810 22:38:03.216513 5096 heartbeater.cc:464] Registering TS with master...
I20260810 22:38:03.217144 5096 heartbeater.cc:511] Master 127.16.233.126:44791 requested a full tablet report, sending...
I20260810 22:38:03.218354 4735 ts_manager.cc:194] Registered new tserver with Master: cb8a097762ea4a968eaedf8d360cf1cb (127.16.233.66:37561)
I20260810 22:38:03.219276 4735 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.233.66:47839
W20260810 22:38:03.325919 5099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:38:03.326202 5099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:38:03.326243 5099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:38:03.330384 5099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:38:03.330461 5099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:38:03.330518 5099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.67
I20260810 22:38:03.335413 5099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:36745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786401215402979-17317-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786401215402979-17317-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.233.67
--webserver_port=0
--tserver_master_addrs=127.16.233.126:44791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.67
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786401215402979-17317-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:38:03.336527 5099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:38:03.337980 5099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:38:03.348184 5107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:03.348189 5104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:03.348184 5105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:38:03.348649 5099 server_base.cc:1034] running on GCE node
I20260810 22:38:03.349125 5099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:38:03.349845 5099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:38:03.351061 5099 hybrid_clock.cc:648] HybridClock initialized: now 1786401483351042 us; error 30 us; skew 500 ppm
I20260810 22:38:03.353117 5099 webserver.cc:460] Webserver started at http://127.16.233.67:43259/ using document root <none> and password file <none>
I20260810 22:38:03.353719 5099 fs_manager.cc:359] Metadata directory not provided
I20260810 22:38:03.353818 5099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:38:03.354068 5099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:38:03.356225 5099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786401215402979-17317-0/minicluster-data/ts-2/data/instance:
uuid: "7fab1fa7d8aa41c793b677114ba3d8ab"
format_stamp: "Formatted at 2026-08-10 22:38:03 on dist-test-slave-77v9"
server_key: "182bbe578849ce1949ed1a4214ec3a02"
server_key_iv: "fc4b77a40e1b6a2b93d09e69268623fa"
server_key_version: "encryptionkey@0"
I20260810 22:38:03.356894 5099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786401215402979-17317-0/minicluster-data/ts-2/wal/instance:
uuid: "7fab1fa7d8aa41c793b677114ba3d8ab"
format_stamp: "Formatted at 2026-08-10 22:38:03 on dist-test-slave-77v9"
server_key: "182bbe578849ce1949ed1a4214ec3a02"
server_key_iv: "fc4b77a40e1b6a2b93d09e69268623fa"
server_key_version: "encryptionkey@0"
I20260810 22:38:03.360685 5099 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260810 22:38:03.363250 5113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:03.364593 5099 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:38:03.364758 5099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786401215402979-17317-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786401215402979-17317-0/minicluster-data/ts-2/wal
uuid: "7fab1fa7d8aa41c793b677114ba3d8ab"
format_stamp: "Formatted at 2026-08-10 22:38:03 on dist-test-slave-77v9"
server_key: "182bbe578849ce1949ed1a4214ec3a02"
server_key_iv: "fc4b77a40e1b6a2b93d09e69268623fa"
server_key_version: "encryptionkey@0"
I20260810 22:38:03.364952 5099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786401215402979-17317-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786401215402979-17317-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786401215402979-17317-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:38:03.385869 5099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:38:03.386682 5099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:38:03.386909 5099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:38:03.387607 5099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:38:03.388648 5099 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:38:03.388723 5099 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:03.388803 5099 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:38:03.388865 5099 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260810 22:38:03.398813 5099 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.67:42363
I20260810 22:38:03.398880 5241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.67:42363 every 8 connection(s)
I20260810 22:38:03.399807 5099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
I20260810 22:38:03.405521 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 5099
I20260810 22:38:03.405622 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786401215402979-17317-0/minicluster-data/ts-2/wal/instance
I20260810 22:38:03.405927 17317 external_mini_cluster.cc:1428] Setting key 3201947da263e43363c730683ec61028
I20260810 22:38:03.409878 4778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:03.401869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:44721 (local address 127.16.233.126:44791)
0810 22:38:03.402058 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:03.402062 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:03.403156 (+ 1094us) server_negotiation.cc:407] Connection header received
0810 22:38:03.404582 (+ 1426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:03.404585 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:03.404635 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:03.404718 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:03.405782 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:03.406492 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:03.407852 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:03.408091 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:03.408206 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:03.409371 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:03.409374 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:03.409378 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:38:03.409390 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:03.409418 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:38:03.409427 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:03.409509 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:03.409667 (+ 158us) server_negotiation.cc:299] Negotiation successful
0810 22:38:03.409731 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 22:38:03.410790 5244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:03.402343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:44791 (local address 127.16.233.67:44721)
0810 22:38:03.402992 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:03.403033 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 22:38:03.404335 (+ 1302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:38:03.404952 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:38:03.404958 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:38:03.405042 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:38:03.405552 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:03.405564 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:03.406628 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:03.406632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:03.407724 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:03.407734 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:03.408248 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:03.408252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:03.408433 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:03.409100 (+ 667us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 22:38:03.409101 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:38:03.409143 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 22:38:03.409211 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:38:03.409511 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:38:03.409525 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:38:03.409531 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 22:38:03.409667 (+ 136us) client_negotiation.cc:240] Negotiation successful
0810 22:38:03.409956 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":115,"threads_started":1}
I20260810 22:38:03.411976 5242 heartbeater.cc:347] Connected to a master server at 127.16.233.126:44791
I20260810 22:38:03.412216 5242 heartbeater.cc:464] Registering TS with master...
I20260810 22:38:03.412755 5242 heartbeater.cc:511] Master 127.16.233.126:44791 requested a full tablet report, sending...
I20260810 22:38:03.413834 4735 ts_manager.cc:194] Registered new tserver with Master: 7fab1fa7d8aa41c793b677114ba3d8ab (127.16.233.67:42363)
I20260810 22:38:03.414379 4735 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.233.67:44721
I20260810 22:38:03.419986 17317 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 22:38:03.426183 4778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:03.422099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38560 (local address 127.16.233.126:44791)
0810 22:38:03.422247 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:03.422254 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:03.422440 (+ 186us) server_negotiation.cc:407] Connection header received
0810 22:38:03.422604 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:03.422608 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:03.422656 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:03.422753 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:03.423086 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:03.423618 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:03.424472 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:03.424720 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:03.424850 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:03.425654 (+ 804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:03.425658 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:03.425661 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:38:03.425675 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:03.425703 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:38:03.425712 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:03.425792 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:03.425976 (+ 184us) server_negotiation.cc:299] Negotiation successful
0810 22:38:03.426036 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 22:38:03.430442 4735 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38560:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 22:38:03.434741 4735 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:38:03.450740 5256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:03.444164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34440 (local address 127.16.233.67:42363)
0810 22:38:03.445476 (+ 1312us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:03.445491 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:03.445515 (+ 24us) server_negotiation.cc:407] Connection header received
0810 22:38:03.445590 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:03.445595 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:03.445795 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:03.446235 (+ 440us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:38:03.446397 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:03.447647 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:03.449404 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:03.449942 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:03.450166 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:03.450399 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:03.450471 (+ 72us) server_negotiation.cc:299] Negotiation successful
0810 22:38:03.450568 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1199,"thread_start_us":122,"threads_started":1}
I20260810 22:38:03.450740 5254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:03.443991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.67:42363 (local address 127.0.0.1:34440)
0810 22:38:03.444376 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:03.444423 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 22:38:03.444553 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:38:03.445982 (+ 1429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:38:03.445988 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:38:03.446013 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:38:03.446253 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:03.446263 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:03.447791 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:03.447795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:03.449248 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:03.449259 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:03.450172 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:03.450175 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:03.450291 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:03.450330 (+ 39us) client_negotiation.cc:769] Sending connection context
0810 22:38:03.450434 (+ 104us) client_negotiation.cc:240] Negotiation successful
0810 22:38:03.450567 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"mutex_wait_us":208,"spinlock_wait_cycles":2816,"thread_start_us":111,"threads_started":1}
I20260810 22:38:03.451491 5257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:03.444246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.65:34753 (local address 127.0.0.1:59054)
0810 22:38:03.444578 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:03.444593 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:38:03.444693 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:38:03.446171 (+ 1478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:38:03.446174 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:38:03.446191 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:38:03.446318 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:03.446326 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:03.447791 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:03.447795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:03.449250 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:03.449259 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:03.450531 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:03.450535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:03.450715 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:03.450769 (+ 54us) client_negotiation.cc:769] Sending connection context
0810 22:38:03.451269 (+ 500us) client_negotiation.cc:240] Negotiation successful
0810 22:38:03.451352 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":91,"thread_start_us":114,"threads_started":1}
I20260810 22:38:03.451610 5260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:03.444900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59054 (local address 127.16.233.65:34753)
0810 22:38:03.445530 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:03.445544 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:03.445567 (+ 23us) server_negotiation.cc:407] Connection header received
0810 22:38:03.445650 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:03.445656 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:03.445847 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:03.446012 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:38:03.446421 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:03.447647 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:03.449385 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:03.450170 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:03.450269 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:03.450458 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:03.451344 (+ 886us) server_negotiation.cc:299] Negotiation successful
0810 22:38:03.451453 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":170,"threads_started":1}
I20260810 22:38:03.451952 5259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:03.444775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.66:37561 (local address 127.0.0.1:55796)
0810 22:38:03.445138 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:03.445152 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:38:03.445247 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:38:03.446024 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:38:03.446027 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:38:03.446054 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:38:03.446288 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:03.446298 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:03.447430 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:03.447434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:03.449134 (+ 1700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:03.449144 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:03.450531 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:03.450535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:03.450712 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:03.450768 (+ 56us) client_negotiation.cc:769] Sending connection context
0810 22:38:03.450843 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 22:38:03.451799 (+ 956us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":189,"spinlock_wait_cycles":768,"thread_start_us":146,"threads_started":1}
I20260810 22:38:03.452448 5261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:03.445084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55796 (local address 127.16.233.66:37561)
0810 22:38:03.445588 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:03.445600 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:03.445618 (+ 18us) server_negotiation.cc:407] Connection header received
0810 22:38:03.445682 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:03.445686 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:03.445853 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:03.446434 (+ 581us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:38:03.446564 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:03.447292 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:03.449275 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:03.450150 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:03.450290 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:03.450458 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:03.452228 (+ 1770us) server_negotiation.cc:299] Negotiation successful
0810 22:38:03.452325 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":129,"threads_started":1}
I20260810 22:38:03.454278 5169 tablet_service.cc:1467] Processing CreateTablet for tablet 6d5ab19a77b54432b944d05ca3a9a91a (DEFAULT_TABLE table=test-table [id=c7e6b72046ec40dc8315cd18fbb3cc05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:38:03.455293 5169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d5ab19a77b54432b944d05ca3a9a91a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:38:03.455201 5020 tablet_service.cc:1467] Processing CreateTablet for tablet 6d5ab19a77b54432b944d05ca3a9a91a (DEFAULT_TABLE table=test-table [id=c7e6b72046ec40dc8315cd18fbb3cc05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:38:03.455353 4873 tablet_service.cc:1467] Processing CreateTablet for tablet 6d5ab19a77b54432b944d05ca3a9a91a (DEFAULT_TABLE table=test-table [id=c7e6b72046ec40dc8315cd18fbb3cc05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:38:03.456233 5020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d5ab19a77b54432b944d05ca3a9a91a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:38:03.456233 4873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d5ab19a77b54432b944d05ca3a9a91a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:38:03.461591 5262 tablet_bootstrap.cc:492] T 6d5ab19a77b54432b944d05ca3a9a91a P 7fab1fa7d8aa41c793b677114ba3d8ab: Bootstrap starting.
I20260810 22:38:03.463662 5263 tablet_bootstrap.cc:492] T 6d5ab19a77b54432b944d05ca3a9a91a P cb8a097762ea4a968eaedf8d360cf1cb: Bootstrap starting.
I20260810 22:38:03.465186 5264 tablet_bootstrap.cc:492] T 6d5ab19a77b54432b944d05ca3a9a91a P d2c6ac572bb54b4192a32044af5ed996: Bootstrap starting.
I20260810 22:38:03.465493 5262 tablet_bootstrap.cc:654] T 6d5ab19a77b54432b944d05ca3a9a91a P 7fab1fa7d8aa41c793b677114ba3d8ab: Neither blocks nor log segments found. Creating new log.
I20260810 22:38:03.465793 5263 tablet_bootstrap.cc:654] T 6d5ab19a77b54432b944d05ca3a9a91a P cb8a097762ea4a968eaedf8d360cf1cb: Neither blocks nor log segments found. Creating new log.
I20260810 22:38:03.466428 5262 log.cc:824] T 6d5ab19a77b54432b944d05ca3a9a91a P 7fab1fa7d8aa41c793b677114ba3d8ab: Log is configured to *not* fsync() on all Append() calls
I20260810 22:38:03.466552 5263 log.cc:824] T 6d5ab19a77b54432b944d05ca3a9a91a P cb8a097762ea4a968eaedf8d360cf1cb: Log is configured to *not* fsync() on all Append() calls
I20260810 22:38:03.467135 5264 tablet_bootstrap.cc:654] T 6d5ab19a77b54432b944d05ca3a9a91a P d2c6ac572bb54b4192a32044af5ed996: Neither blocks nor log segments found. Creating new log.
I20260810 22:38:03.467979 5264 log.cc:824] T 6d5ab19a77b54432b944d05ca3a9a91a P d2c6ac572bb54b4192a32044af5ed996: Log is configured to *not* fsync() on all Append() calls
I20260810 22:38:03.468801 5262 tablet_bootstrap.cc:492] T 6d5ab19a77b54432b944d05ca3a9a91a P 7fab1fa7d8aa41c793b677114ba3d8ab: No bootstrap required, opened a new log
I20260810 22:38:03.469084 5262 ts_tablet_manager.cc:1397] T 6d5ab19a77b54432b944d05ca3a9a91a P 7fab1fa7d8aa41c793b677114ba3d8ab: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260810 22:38:03.469514 5263 tablet_bootstrap.cc:492] T 6d5ab19a77b54432b944d05ca3a9a91a P cb8a097762ea4a968eaedf8d360cf1cb: No bootstrap required, opened a new log
I20260810 22:38:03.469722 5263 ts_tablet_manager.cc:1397] T 6d5ab19a77b54432b944d05ca3a9a91a P cb8a097762ea4a968eaedf8d360cf1cb: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 22:38:03.470152 5264 tablet_bootstrap.cc:492] T 6d5ab19a77b54432b944d05ca3a9a91a P d2c6ac572bb54b4192a32044af5ed996: No bootstrap required, opened a new log
I20260810 22:38:03.470361 5264 ts_tablet_manager.cc:1397] T 6d5ab19a77b54432b944d05ca3a9a91a P d2c6ac572bb54b4192a32044af5ed996: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260810 22:38:03.473580 5264 raft_consensus.cc:348] T 6d5ab19a77b54432b944d05ca3a9a91a P d2c6ac572bb54b4192a32044af5ed996 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fab1fa7d8aa41c793b677114ba3d8ab" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 42363 } } peers { permanent_uuid: "cb8a097762ea4a968eaedf8d360cf1cb" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 37561 } } peers { permanent_uuid: "d2c6ac572bb54b4192a32044af5ed996" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 34753 } }
I20260810 22:38:03.473618 5262 raft_consensus.cc:348] T 6d5ab19a77b54432b944d05ca3a9a91a P 7fab1fa7d8aa41c793b677114ba3d8ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fab1fa7d8aa41c793b677114ba3d8ab" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 42363 } } peers { permanent_uuid: "cb8a097762ea4a968eaedf8d360cf1cb" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 37561 } } peers { permanent_uuid: "d2c6ac572bb54b4192a32044af5ed996" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 34753 } }
I20260810 22:38:03.473618 5263 raft_consensus.cc:348] T 6d5ab19a77b54432b944d05ca3a9a91a P cb8a097762ea4a968eaedf8d360cf1cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fab1fa7d8aa41c793b677114ba3d8ab" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 42363 } } peers { permanent_uuid: "cb8a097762ea4a968eaedf8d360cf1cb" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 37561 } } peers { permanent_uuid: "d2c6ac572bb54b4192a32044af5ed996" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 34753 } }
I20260810 22:38:03.473850 5264 raft_consensus.cc:374] T 6d5ab19a77b54432b944d05ca3a9a91a P d2c6ac572bb54b4192a32044af5ed996 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:38:03.473872 5263 raft_consensus.cc:374] T 6d5ab19a77b54432b944d05ca3a9a91a P cb8a097762ea4a968eaedf8d360cf1cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:38:03.473872 5262 raft_consensus.cc:374] T 6d5ab19a77b54432b944d05ca3a9a91a P 7fab1fa7d8aa41c793b677114ba3d8ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:38:03.473914 5264 raft_consensus.cc:729] T 6d5ab19a77b54432b944d05ca3a9a91a P d2c6ac572bb54b4192a32044af5ed996 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2c6ac572bb54b4192a32044af5ed996, State: Initialized, Role: FOLLOWER
I20260810 22:38:03.473925 5263 raft_consensus.cc:729] T 6d5ab19a77b54432b944d05ca3a9a91a P cb8a097762ea4a968eaedf8d360cf1cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb8a097762ea4a968eaedf8d360cf1cb, State: Initialized, Role: FOLLOWER
I20260810 22:38:03.473934 5262 raft_consensus.cc:729] T 6d5ab19a77b54432b944d05ca3a9a91a P 7fab1fa7d8aa41c793b677114ba3d8ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fab1fa7d8aa41c793b677114ba3d8ab, State: Initialized, Role: FOLLOWER
I20260810 22:38:03.474383 5264 consensus_queue.cc:260] T 6d5ab19a77b54432b944d05ca3a9a91a P d2c6ac572bb54b4192a32044af5ed996 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fab1fa7d8aa41c793b677114ba3d8ab" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 42363 } } peers { permanent_uuid: "cb8a097762ea4a968eaedf8d360cf1cb" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 37561 } } peers { permanent_uuid: "d2c6ac572bb54b4192a32044af5ed996" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 34753 } }
I20260810 22:38:03.474381 5263 consensus_queue.cc:260] T 6d5ab19a77b54432b944d05ca3a9a91a P cb8a097762ea4a968eaedf8d360cf1cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fab1fa7d8aa41c793b677114ba3d8ab" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 42363 } } peers { permanent_uuid: "cb8a097762ea4a968eaedf8d360cf1cb" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 37561 } } peers { permanent_uuid: "d2c6ac572bb54b4192a32044af5ed996" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 34753 } }
I20260810 22:38:03.474381 5262 consensus_queue.cc:260] T 6d5ab19a77b54432b944d05ca3a9a91a P 7fab1fa7d8aa41c793b677114ba3d8ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fab1fa7d8aa41c793b677114ba3d8ab" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 42363 } } peers { permanent_uuid: "cb8a097762ea4a968eaedf8d360cf1cb" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 37561 } } peers { permanent_uuid: "d2c6ac572bb54b4192a32044af5ed996" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 34753 } }
I20260810 22:38:03.475462 5262 ts_tablet_manager.cc:1428] T 6d5ab19a77b54432b944d05ca3a9a91a P 7fab1fa7d8aa41c793b677114ba3d8ab: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 22:38:03.475462 5264 ts_tablet_manager.cc:1428] T 6d5ab19a77b54432b944d05ca3a9a91a P d2c6ac572bb54b4192a32044af5ed996: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 22:38:03.475462 5263 ts_tablet_manager.cc:1428] T 6d5ab19a77b54432b944d05ca3a9a91a P cb8a097762ea4a968eaedf8d360cf1cb: Time spent starting tablet: real 0.006s user 0.000s sys 0.005s
I20260810 22:38:03.475826 4947 heartbeater.cc:503] Master 127.16.233.126:44791 was elected leader, sending a full tablet report...
I20260810 22:38:03.475833 5242 heartbeater.cc:503] Master 127.16.233.126:44791 was elected leader, sending a full tablet report...
I20260810 22:38:03.476663 5096 heartbeater.cc:503] Master 127.16.233.126:44791 was elected leader, sending a full tablet report...
W20260810 22:38:03.512182 4948 tablet.cc:2369] T 6d5ab19a77b54432b944d05ca3a9a91a P d2c6ac572bb54b4192a32044af5ed996: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 22:38:03.650905 5243 tablet.cc:2369] T 6d5ab19a77b54432b944d05ca3a9a91a P 7fab1fa7d8aa41c793b677114ba3d8ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:38:03.676167 5270 raft_consensus.cc:482] T 6d5ab19a77b54432b944d05ca3a9a91a P 7fab1fa7d8aa41c793b677114ba3d8ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:38:03.676406 5270 raft_consensus.cc:504] T 6d5ab19a77b54432b944d05ca3a9a91a P 7fab1fa7d8aa41c793b677114ba3d8ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fab1fa7d8aa41c793b677114ba3d8ab" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 42363 } } peers { permanent_uuid: "cb8a097762ea4a968eaedf8d360cf1cb" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 37561 } } peers { permanent_uuid: "d2c6ac572bb54b4192a32044af5ed996" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 34753 } }
I20260810 22:38:03.677714 5270 leader_election.cc:288] T 6d5ab19a77b54432b944d05ca3a9a91a P 7fab1fa7d8aa41c793b677114ba3d8ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb8a097762ea4a968eaedf8d360cf1cb (127.16.233.66:37561), d2c6ac572bb54b4192a32044af5ed996 (127.16.233.65:34753)
I20260810 22:38:03.682037 5244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:03.677970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.66:37561 (local address 127.16.233.67:36263)
0810 22:38:03.678200 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:03.678216 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 22:38:03.678324 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:38:03.678629 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:38:03.678632 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:38:03.678650 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:38:03.678837 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:03.678844 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:03.679604 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:03.679607 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:03.680436 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:03.680445 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:03.681444 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:03.681448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:03.681667 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:03.681709 (+ 42us) client_negotiation.cc:769] Sending connection context
0810 22:38:03.681792 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 22:38:03.681884 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":82}
I20260810 22:38:03.682065 5261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:03.678059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:36263 (local address 127.16.233.66:37561)
0810 22:38:03.678226 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:03.678230 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:03.678263 (+ 33us) server_negotiation.cc:407] Connection header received
0810 22:38:03.678450 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:03.678452 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:03.678500 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:03.678612 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:38:03.678962 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:03.679487 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:03.680566 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:03.681163 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:03.681320 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:03.681397 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:03.681825 (+ 428us) server_negotiation.cc:299] Negotiation successful
0810 22:38:03.681904 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 22:38:03.682152 5260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:03.678107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:38623 (local address 127.16.233.65:34753)
0810 22:38:03.678232 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:03.678238 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:03.678556 (+ 318us) server_negotiation.cc:407] Connection header received
0810 22:38:03.678710 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:03.678712 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:03.678757 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:03.678844 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:38:03.679152 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:03.679657 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:03.680684 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:03.681283 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:03.681432 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:03.681479 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:03.681938 (+ 459us) server_negotiation.cc:299] Negotiation successful
0810 22:38:03.682022 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 22:38:03.682477 5273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:03.677992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.65:34753 (local address 127.16.233.67:38623)
0810 22:38:03.678455 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:03.678467 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 22:38:03.678581 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:38:03.678855 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:38:03.678858 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:38:03.678875 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:38:03.679025 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:03.679031 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:03.679772 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:03.679778 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:03.680552 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:03.680560 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:03.681420 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:03.681433 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:03.681662 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:03.681706 (+ 44us) client_negotiation.cc:769] Sending connection context
0810 22:38:03.682295 (+ 589us) client_negotiation.cc:240] Negotiation successful
0810 22:38:03.682373 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"mutex_wait_us":51,"thread_start_us":128,"threads_started":1}
I20260810 22:38:03.683058 5043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d5ab19a77b54432b944d05ca3a9a91a" candidate_uuid: "7fab1fa7d8aa41c793b677114ba3d8ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb8a097762ea4a968eaedf8d360cf1cb" is_pre_election: true
I20260810 22:38:03.683156 4895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d5ab19a77b54432b944d05ca3a9a91a" candidate_uuid: "7fab1fa7d8aa41c793b677114ba3d8ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2c6ac572bb54b4192a32044af5ed996" is_pre_election: true
I20260810 22:38:03.683441 5043 raft_consensus.cc:2454] T 6d5ab19a77b54432b944d05ca3a9a91a P cb8a097762ea4a968eaedf8d360cf1cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fab1fa7d8aa41c793b677114ba3d8ab in term 0.
I20260810 22:38:03.683444 4895 raft_consensus.cc:2454] T 6d5ab19a77b54432b944d05ca3a9a91a P d2c6ac572bb54b4192a32044af5ed996 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fab1fa7d8aa41c793b677114ba3d8ab in term 0.
I20260810 22:38:03.684024 5115 leader_election.cc:302] T 6d5ab19a77b54432b944d05ca3a9a91a P 7fab1fa7d8aa41c793b677114ba3d8ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7fab1fa7d8aa41c793b677114ba3d8ab, cb8a097762ea4a968eaedf8d360cf1cb, d2c6ac572bb54b4192a32044af5ed996; no voters:
I20260810 22:38:03.684346 5270 raft_consensus.cc:2781] T 6d5ab19a77b54432b944d05ca3a9a91a P 7fab1fa7d8aa41c793b677114ba3d8ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:38:03.684440 5270 raft_consensus.cc:482] T 6d5ab19a77b54432b944d05ca3a9a91a P 7fab1fa7d8aa41c793b677114ba3d8ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:38:03.684484 5270 raft_consensus.cc:3037] T 6d5ab19a77b54432b944d05ca3a9a91a P 7fab1fa7d8aa41c793b677114ba3d8ab [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:38:03.685644 5270 raft_consensus.cc:504] T 6d5ab19a77b54432b944d05ca3a9a91a P 7fab1fa7d8aa41c793b677114ba3d8ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fab1fa7d8aa41c793b677114ba3d8ab" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 42363 } } peers { permanent_uuid: "cb8a097762ea4a968eaedf8d360cf1cb" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 37561 } } peers { permanent_uuid: "d2c6ac572bb54b4192a32044af5ed996" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 34753 } }
I20260810 22:38:03.685999 5270 leader_election.cc:288] T 6d5ab19a77b54432b944d05ca3a9a91a P 7fab1fa7d8aa41c793b677114ba3d8ab [CANDIDATE]: Term 1 election: Requested vote from peers cb8a097762ea4a968eaedf8d360cf1cb (127.16.233.66:37561), d2c6ac572bb54b4192a32044af5ed996 (127.16.233.65:34753)
I20260810 22:38:03.686367 5043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d5ab19a77b54432b944d05ca3a9a91a" candidate_uuid: "7fab1fa7d8aa41c793b677114ba3d8ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb8a097762ea4a968eaedf8d360cf1cb"
I20260810 22:38:03.686401 4895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d5ab19a77b54432b944d05ca3a9a91a" candidate_uuid: "7fab1fa7d8aa41c793b677114ba3d8ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2c6ac572bb54b4192a32044af5ed996"
I20260810 22:38:03.686525 5043 raft_consensus.cc:3037] T 6d5ab19a77b54432b944d05ca3a9a91a P cb8a097762ea4a968eaedf8d360cf1cb [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:38:03.686609 4895 raft_consensus.cc:3037] T 6d5ab19a77b54432b944d05ca3a9a91a P d2c6ac572bb54b4192a32044af5ed996 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:38:03.687711 5043 raft_consensus.cc:2454] T 6d5ab19a77b54432b944d05ca3a9a91a P cb8a097762ea4a968eaedf8d360cf1cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fab1fa7d8aa41c793b677114ba3d8ab in term 1.
I20260810 22:38:03.687731 4895 raft_consensus.cc:2454] T 6d5ab19a77b54432b944d05ca3a9a91a P d2c6ac572bb54b4192a32044af5ed996 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fab1fa7d8aa41c793b677114ba3d8ab in term 1.
I20260810 22:38:03.688092 5117 leader_election.cc:302] T 6d5ab19a77b54432b944d05ca3a9a91a P 7fab1fa7d8aa41c793b677114ba3d8ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fab1fa7d8aa41c793b677114ba3d8ab, cb8a097762ea4a968eaedf8d360cf1cb; no voters:
I20260810 22:38:03.688292 5270 raft_consensus.cc:2781] T 6d5ab19a77b54432b944d05ca3a9a91a P 7fab1fa7d8aa41c793b677114ba3d8ab [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:38:03.688526 5270 raft_consensus.cc:686] T 6d5ab19a77b54432b944d05ca3a9a91a P 7fab1fa7d8aa41c793b677114ba3d8ab [term 1 LEADER]: Becoming Leader. State: Replica: 7fab1fa7d8aa41c793b677114ba3d8ab, State: Running, Role: LEADER
I20260810 22:38:03.688958 5270 consensus_queue.cc:237] T 6d5ab19a77b54432b944d05ca3a9a91a P 7fab1fa7d8aa41c793b677114ba3d8ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fab1fa7d8aa41c793b677114ba3d8ab" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 42363 } } peers { permanent_uuid: "cb8a097762ea4a968eaedf8d360cf1cb" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 37561 } } peers { permanent_uuid: "d2c6ac572bb54b4192a32044af5ed996" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 34753 } }
I20260810 22:38:03.692615 4735 catalog_manager.cc:5495] T 6d5ab19a77b54432b944d05ca3a9a91a P 7fab1fa7d8aa41c793b677114ba3d8ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 7fab1fa7d8aa41c793b677114ba3d8ab (127.16.233.67). New cstate: current_term: 1 leader_uuid: "7fab1fa7d8aa41c793b677114ba3d8ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fab1fa7d8aa41c793b677114ba3d8ab" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 42363 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb8a097762ea4a968eaedf8d360cf1cb" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 37561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2c6ac572bb54b4192a32044af5ed996" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 34753 } health_report { overall_health: UNKNOWN } } }
W20260810 22:38:03.704567 5097 tablet.cc:2369] T 6d5ab19a77b54432b944d05ca3a9a91a P cb8a097762ea4a968eaedf8d360cf1cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:38:03.748117 5256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:03.744133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34448 (local address 127.16.233.67:42363)
0810 22:38:03.744310 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:03.744318 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:03.744337 (+ 19us) server_negotiation.cc:407] Connection header received
0810 22:38:03.744449 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:03.744454 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:03.744565 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:03.744686 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:38:03.744977 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:03.745539 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:03.746461 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:03.746687 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:03.746787 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:03.747110 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:38:03.747569 (+ 459us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:38:03.747662 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:03.747759 (+ 97us) server_negotiation.cc:299] Negotiation successful
0810 22:38:03.747903 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260810 22:38:03.754637 5043 raft_consensus.cc:1261] T 6d5ab19a77b54432b944d05ca3a9a91a P cb8a097762ea4a968eaedf8d360cf1cb [term 1 FOLLOWER]: Refusing update from remote peer 7fab1fa7d8aa41c793b677114ba3d8ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:38:03.754649 4895 raft_consensus.cc:1261] T 6d5ab19a77b54432b944d05ca3a9a91a P d2c6ac572bb54b4192a32044af5ed996 [term 1 FOLLOWER]: Refusing update from remote peer 7fab1fa7d8aa41c793b677114ba3d8ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:38:03.755409 5274 consensus_queue.cc:1035] T 6d5ab19a77b54432b944d05ca3a9a91a P 7fab1fa7d8aa41c793b677114ba3d8ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2c6ac572bb54b4192a32044af5ed996" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 34753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:38:03.755625 5270 consensus_queue.cc:1035] T 6d5ab19a77b54432b944d05ca3a9a91a P 7fab1fa7d8aa41c793b677114ba3d8ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb8a097762ea4a968eaedf8d360cf1cb" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 37561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:38:03.764104 5277 mvcc.cc:205] Tried to move back new op lower bound from 7317100477450137600 to 7317100477193138176. Current Snapshot: MvccSnapshot[applied={T|T < 7317100477450137600}]
I20260810 22:38:03.767568 5279 mvcc.cc:205] Tried to move back new op lower bound from 7317100477450137600 to 7317100477193138176. Current Snapshot: MvccSnapshot[applied={T|T < 7317100477450137600}]
I20260810 22:38:03.772600 4735 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:38:03.772851 4735 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:38:03.775663 4735 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6aad93b30525408ea2810a23d883461a: Sending DeleteTablet for 3 replicas of tablet 6d5ab19a77b54432b944d05ca3a9a91a
I20260810 22:38:03.776693 4873 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d5ab19a77b54432b944d05ca3a9a91a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:38:03 UTC) from {username='slave'} at 127.0.0.1:59054
I20260810 22:38:03.776734 5020 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d5ab19a77b54432b944d05ca3a9a91a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:38:03 UTC) from {username='slave'} at 127.0.0.1:55796
I20260810 22:38:03.777372 5288 tablet_replica.cc:330] T 6d5ab19a77b54432b944d05ca3a9a91a P d2c6ac572bb54b4192a32044af5ed996: stopping tablet replica
I20260810 22:38:03.777390 5169 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d5ab19a77b54432b944d05ca3a9a91a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:38:03 UTC) from {username='slave'} at 127.0.0.1:34440
I20260810 22:38:03.777724 5289 tablet_replica.cc:330] T 6d5ab19a77b54432b944d05ca3a9a91a P cb8a097762ea4a968eaedf8d360cf1cb: stopping tablet replica
I20260810 22:38:03.777873 5290 tablet_replica.cc:330] T 6d5ab19a77b54432b944d05ca3a9a91a P 7fab1fa7d8aa41c793b677114ba3d8ab: stopping tablet replica
I20260810 22:38:03.778210 5289 raft_consensus.cc:2229] T 6d5ab19a77b54432b944d05ca3a9a91a P cb8a097762ea4a968eaedf8d360cf1cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:38:03.778210 5290 raft_consensus.cc:2229] T 6d5ab19a77b54432b944d05ca3a9a91a P 7fab1fa7d8aa41c793b677114ba3d8ab [term 1 LEADER]: Raft consensus shutting down.
I20260810 22:38:03.778342 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 4800
I20260810 22:38:03.778604 5290 raft_consensus.cc:2258] T 6d5ab19a77b54432b944d05ca3a9a91a P 7fab1fa7d8aa41c793b677114ba3d8ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:38:03.778587 5288 raft_consensus.cc:2229] T 6d5ab19a77b54432b944d05ca3a9a91a P d2c6ac572bb54b4192a32044af5ed996 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:38:03.779264 5289 raft_consensus.cc:2258] T 6d5ab19a77b54432b944d05ca3a9a91a P cb8a097762ea4a968eaedf8d360cf1cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:38:03.782817 5290 ts_tablet_manager.cc:1905] T 6d5ab19a77b54432b944d05ca3a9a91a P 7fab1fa7d8aa41c793b677114ba3d8ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:38:03.785548 5290 ts_tablet_manager.cc:1918] T 6d5ab19a77b54432b944d05ca3a9a91a P 7fab1fa7d8aa41c793b677114ba3d8ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:38:03.785691 5290 log.cc:1196] T 6d5ab19a77b54432b944d05ca3a9a91a P 7fab1fa7d8aa41c793b677114ba3d8ab: Deleting WAL directory at /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786401215402979-17317-0/minicluster-data/ts-2/wal/wals/6d5ab19a77b54432b944d05ca3a9a91a
I20260810 22:38:03.786006 5290 ts_tablet_manager.cc:1939] T 6d5ab19a77b54432b944d05ca3a9a91a P 7fab1fa7d8aa41c793b677114ba3d8ab: Deleting consensus metadata
I20260810 22:38:03.787361 4721 catalog_manager.cc:4841] TS 7fab1fa7d8aa41c793b677114ba3d8ab (127.16.233.67:42363): tablet 6d5ab19a77b54432b944d05ca3a9a91a (table test-table [id=c7e6b72046ec40dc8315cd18fbb3cc05]) successfully deleted
I20260810 22:38:03.787708 5289 ts_tablet_manager.cc:1905] T 6d5ab19a77b54432b944d05ca3a9a91a P cb8a097762ea4a968eaedf8d360cf1cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:38:03.789976 5289 ts_tablet_manager.cc:1918] T 6d5ab19a77b54432b944d05ca3a9a91a P cb8a097762ea4a968eaedf8d360cf1cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:38:03.790107 5289 log.cc:1196] T 6d5ab19a77b54432b944d05ca3a9a91a P cb8a097762ea4a968eaedf8d360cf1cb: Deleting WAL directory at /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786401215402979-17317-0/minicluster-data/ts-1/wal/wals/6d5ab19a77b54432b944d05ca3a9a91a
I20260810 22:38:03.790463 5289 ts_tablet_manager.cc:1939] T 6d5ab19a77b54432b944d05ca3a9a91a P cb8a097762ea4a968eaedf8d360cf1cb: Deleting consensus metadata
W20260810 22:38:03.790650 4718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.233.65:34753 (error 108)
I20260810 22:38:03.791760 4721 catalog_manager.cc:4841] TS cb8a097762ea4a968eaedf8d360cf1cb (127.16.233.66:37561): tablet 6d5ab19a77b54432b944d05ca3a9a91a (table test-table [id=c7e6b72046ec40dc8315cd18fbb3cc05]) successfully deleted
I20260810 22:38:03.791949 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 4950
I20260810 22:38:03.792358 5259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:03.791986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.65:34753 (local address 127.0.0.1:59066)
0810 22:38:03.792172 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:03.792262 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.65:34753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
W20260810 22:38:03.794932 4718 catalog_manager.cc:4584] TS d2c6ac572bb54b4192a32044af5ed996 (127.16.233.65:34753): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6d5ab19a77b54432b944d05ca3a9a91a: Network error: Client connection negotiation failed: client connection to 127.16.233.65:34753: connect: Connection refused (error 111)
I20260810 22:38:03.803016 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 5099
I20260810 22:38:03.810786 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 4698
2026-08-10T22:38:03Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1320 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:38:03 dist-test-slave-77v9 krb5kdc[5294](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:38:03 dist-test-slave-77v9 krb5kdc[5294](info): set up 2 sockets
Aug 10 22:38:03 dist-test-slave-77v9 krb5kdc[5294](info): commencing operation
krb5kdc: starting...
W20260810 22:38:04.948100 17317 mini_kdc.cc:121] Time spent starting KDC: real 1.112s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Aug 10 22:38:05 dist-test-slave-77v9 krb5kdc[5294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:38:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:38:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.233.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab.
Entry for principal kudu/127.16.233.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab.
I20260810 22:38:05.097936 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:38213
--webserver_interface=127.16.233.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab
--principal=kudu/127.16.233.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:37931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.126:38213
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:38:05.226828 5309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:38:05.227159 5309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:38:05.227205 5309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:38:05.231335 5309 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:38:05.231418 5309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:38:05.231438 5309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:38:05.231456 5309 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:38:05.231472 5309 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:38:05.237323 5309 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:37931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.126:38213
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab
--principal=kudu/127.16.233.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:38213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:38:05.238487 5309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:38:05.239513 5309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:38:05.248695 5315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:05.248715 5319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:05.248700 5317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:38:05.249203 5309 server_base.cc:1034] running on GCE node
I20260810 22:38:05.250077 5309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:38:05.251425 5309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:38:05.252731 5309 hybrid_clock.cc:648] HybridClock initialized: now 1786401485252701 us; error 56 us; skew 500 ppm
Aug 10 22:38:05 dist-test-slave-77v9 krb5kdc[5294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401485, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:38:05.261127 5309 init.cc:377] Logged in from keytab as kudu/127.16.233.126@KRBTEST.COM (short username kudu)
I20260810 22:38:05.262413 5309 webserver.cc:460] Webserver started at http://127.16.233.126:35201/ using document root <none> and password file <none>
I20260810 22:38:05.263139 5309 fs_manager.cc:359] Metadata directory not provided
I20260810 22:38:05.263279 5309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:38:05.263552 5309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:38:05.265951 5309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/minicluster-data/master-0/data/instance:
uuid: "5b8905fabd214d7b8b7294fe3c6f2fea"
format_stamp: "Formatted at 2026-08-10 22:38:05 on dist-test-slave-77v9"
server_key: "a18c54efe92df086af1448944c6356a9"
server_key_iv: "cc9ac37f1db851ffb56e3f9521bd71eb"
server_key_version: "encryptionkey@0"
I20260810 22:38:05.266601 5309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/minicluster-data/master-0/wal/instance:
uuid: "5b8905fabd214d7b8b7294fe3c6f2fea"
format_stamp: "Formatted at 2026-08-10 22:38:05 on dist-test-slave-77v9"
server_key: "a18c54efe92df086af1448944c6356a9"
server_key_iv: "cc9ac37f1db851ffb56e3f9521bd71eb"
server_key_version: "encryptionkey@0"
I20260810 22:38:05.270937 5309 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 22:38:05.273765 5326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:05.275221 5309 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 22:38:05.275410 5309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/minicluster-data/master-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/minicluster-data/master-0/wal
uuid: "5b8905fabd214d7b8b7294fe3c6f2fea"
format_stamp: "Formatted at 2026-08-10 22:38:05 on dist-test-slave-77v9"
server_key: "a18c54efe92df086af1448944c6356a9"
server_key_iv: "cc9ac37f1db851ffb56e3f9521bd71eb"
server_key_version: "encryptionkey@0"
I20260810 22:38:05.275561 5309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:38:05.298884 5309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:38:05.302999 5309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:38:05.303256 5309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:38:05.311715 5309 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.126:38213
I20260810 22:38:05.311836 5386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.126:38213 every 8 connection(s)
I20260810 22:38:05.312903 5309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
I20260810 22:38:05.316022 5387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:38:05.317500 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 5309
I20260810 22:38:05.317665 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/minicluster-data/master-0/wal/instance
I20260810 22:38:05.318037 17317 external_mini_cluster.cc:1428] Setting key 8ba67ec5c307daac853e62be66497c83
I20260810 22:38:05.322757 5387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b8905fabd214d7b8b7294fe3c6f2fea: Bootstrap starting.
I20260810 22:38:05.325255 5387 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b8905fabd214d7b8b7294fe3c6f2fea: Neither blocks nor log segments found. Creating new log.
I20260810 22:38:05.326053 5387 log.cc:824] T 00000000000000000000000000000000 P 5b8905fabd214d7b8b7294fe3c6f2fea: Log is configured to *not* fsync() on all Append() calls
Aug 10 22:38:05 dist-test-slave-77v9 krb5kdc[5294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:38:05.328336 5387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b8905fabd214d7b8b7294fe3c6f2fea: No bootstrap required, opened a new log
I20260810 22:38:05.332224 5387 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b8905fabd214d7b8b7294fe3c6f2fea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b8905fabd214d7b8b7294fe3c6f2fea" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 38213 } }
I20260810 22:38:05.332525 5387 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b8905fabd214d7b8b7294fe3c6f2fea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:38:05.332631 5387 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b8905fabd214d7b8b7294fe3c6f2fea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b8905fabd214d7b8b7294fe3c6f2fea, State: Initialized, Role: FOLLOWER
I20260810 22:38:05.333220 5387 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b8905fabd214d7b8b7294fe3c6f2fea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b8905fabd214d7b8b7294fe3c6f2fea" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 38213 } }
I20260810 22:38:05.333367 5387 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b8905fabd214d7b8b7294fe3c6f2fea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:38:05.333375 5390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:05.319607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53192 (local address 127.16.233.126:38213)
0810 22:38:05.320159 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:05.320176 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:05.320214 (+ 38us) server_negotiation.cc:407] Connection header received
0810 22:38:05.320936 (+ 722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:05.320955 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:05.321291 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:05.321626 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:05.322515 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:05.323543 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:05.324435 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:05.325014 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:05.325189 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:05.327910 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:05.327915 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:05.327929 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:38:05.327963 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:05.330001 (+ 2038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:05.330603 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:05.330612 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:05.330622 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:05.330740 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:05.331059 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:05.331062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:05.331064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:05.331473 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:05.331685 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:05.332065 (+ 380us) server_negotiation.cc:299] Negotiation successful
0810 22:38:05.332354 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":190,"threads_started":1}
I20260810 22:38:05.333474 5387 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b8905fabd214d7b8b7294fe3c6f2fea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:38:05.333580 5387 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b8905fabd214d7b8b7294fe3c6f2fea [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:38:05.334733 5387 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b8905fabd214d7b8b7294fe3c6f2fea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b8905fabd214d7b8b7294fe3c6f2fea" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 38213 } }
I20260810 22:38:05.335105 5387 leader_election.cc:302] T 00000000000000000000000000000000 P 5b8905fabd214d7b8b7294fe3c6f2fea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b8905fabd214d7b8b7294fe3c6f2fea; no voters:
I20260810 22:38:05.335481 5387 leader_election.cc:288] T 00000000000000000000000000000000 P 5b8905fabd214d7b8b7294fe3c6f2fea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:38:05.335600 5392 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b8905fabd214d7b8b7294fe3c6f2fea [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:38:05.335815 5392 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b8905fabd214d7b8b7294fe3c6f2fea [term 1 LEADER]: Becoming Leader. State: Replica: 5b8905fabd214d7b8b7294fe3c6f2fea, State: Running, Role: LEADER
I20260810 22:38:05.336166 5392 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b8905fabd214d7b8b7294fe3c6f2fea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b8905fabd214d7b8b7294fe3c6f2fea" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 38213 } }
I20260810 22:38:05.336799 5387 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b8905fabd214d7b8b7294fe3c6f2fea [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:38:05.337349 5392 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b8905fabd214d7b8b7294fe3c6f2fea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b8905fabd214d7b8b7294fe3c6f2fea. Latest consensus state: current_term: 1 leader_uuid: "5b8905fabd214d7b8b7294fe3c6f2fea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b8905fabd214d7b8b7294fe3c6f2fea" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 38213 } } }
I20260810 22:38:05.337476 5392 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b8905fabd214d7b8b7294fe3c6f2fea [sys.catalog]: This master's current role is: LEADER
I20260810 22:38:05.337924 5393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b8905fabd214d7b8b7294fe3c6f2fea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b8905fabd214d7b8b7294fe3c6f2fea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b8905fabd214d7b8b7294fe3c6f2fea" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 38213 } } }
I20260810 22:38:05.338027 5393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b8905fabd214d7b8b7294fe3c6f2fea [sys.catalog]: This master's current role is: LEADER
W20260810 22:38:05.342371 5409 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5b8905fabd214d7b8b7294fe3c6f2fea: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 22:38:05.342448 5409 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 22:38:05.342510 5397 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:38:05.343335 5397 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:38:05.349659 5397 catalog_manager.cc:1347] Generated new cluster ID: 738d054b575140fca560f565a9038856
I20260810 22:38:05.349778 5397 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:38:05.363327 5397 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:38:05.364423 5397 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:38:05.371024 5397 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b8905fabd214d7b8b7294fe3c6f2fea: Generated new TSK 0
I20260810 22:38:05.371750 5397 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.233.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab.
Entry for principal kudu/127.16.233.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab.
I20260810 22:38:05.416378 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.65:0
--local_ip_for_outbound_sockets=127.16.233.65
--webserver_interface=127.16.233.65
--webserver_port=0
--tserver_master_addrs=127.16.233.126:38213
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab
--principal=kudu/127.16.233.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:37931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:38:05.541038 5417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:38:05.541332 5417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:38:05.541404 5417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:38:05.545639 5417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:38:05.545709 5417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:38:05.545804 5417 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.65
I20260810 22:38:05.551460 5417 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:37931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab
--principal=kudu/127.16.233.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.233.65
--webserver_port=0
--tserver_master_addrs=127.16.233.126:38213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.65
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:38:05.552676 5417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:38:05.553818 5417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:38:05.564358 5425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:05.564422 5423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:05.564358 5422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:38:05.564595 5417 server_base.cc:1034] running on GCE node
I20260810 22:38:05.565160 5417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:38:05.565928 5417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:38:05.567164 5417 hybrid_clock.cc:648] HybridClock initialized: now 1786401485567128 us; error 48 us; skew 500 ppm
Aug 10 22:38:05 dist-test-slave-77v9 krb5kdc[5294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401485, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:38:05.570353 5417 init.cc:377] Logged in from keytab as kudu/127.16.233.65@KRBTEST.COM (short username kudu)
I20260810 22:38:05.571346 5417 webserver.cc:460] Webserver started at http://127.16.233.65:36371/ using document root <none> and password file <none>
I20260810 22:38:05.571952 5417 fs_manager.cc:359] Metadata directory not provided
I20260810 22:38:05.572059 5417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:38:05.572317 5417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:38:05.574604 5417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/minicluster-data/ts-0/data/instance:
uuid: "178b06fdf44849fe9daac479a7651835"
format_stamp: "Formatted at 2026-08-10 22:38:05 on dist-test-slave-77v9"
server_key: "e6eece847b42d5c13c6ff1d307825f19"
server_key_iv: "d437765a1e340ad8d7ee55ffedf78cf0"
server_key_version: "encryptionkey@0"
I20260810 22:38:05.575228 5417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/minicluster-data/ts-0/wal/instance:
uuid: "178b06fdf44849fe9daac479a7651835"
format_stamp: "Formatted at 2026-08-10 22:38:05 on dist-test-slave-77v9"
server_key: "e6eece847b42d5c13c6ff1d307825f19"
server_key_iv: "d437765a1e340ad8d7ee55ffedf78cf0"
server_key_version: "encryptionkey@0"
I20260810 22:38:05.579914 5417 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260810 22:38:05.582979 5433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:05.584525 5417 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 22:38:05.584717 5417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/minicluster-data/ts-0/wal
uuid: "178b06fdf44849fe9daac479a7651835"
format_stamp: "Formatted at 2026-08-10 22:38:05 on dist-test-slave-77v9"
server_key: "e6eece847b42d5c13c6ff1d307825f19"
server_key_iv: "d437765a1e340ad8d7ee55ffedf78cf0"
server_key_version: "encryptionkey@0"
I20260810 22:38:05.584949 5417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:38:05.609429 5417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:38:05.613132 5417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:38:05.613364 5417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:38:05.614041 5417 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:38:05.614935 5417 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:38:05.614985 5417 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:05.615096 5417 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:38:05.615136 5417 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:05.626467 5417 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.65:34743
I20260810 22:38:05.626502 5552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.65:34743 every 8 connection(s)
I20260810 22:38:05.627449 5417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
I20260810 22:38:05.635775 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 5417
I20260810 22:38:05.635944 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/minicluster-data/ts-0/wal/instance
I20260810 22:38:05.636338 17317 external_mini_cluster.cc:1428] Setting key ccc4e4ae5168ffeb1645dbf92da87533
Aug 10 22:38:05 dist-test-slave-77v9 krb5kdc[5294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401485, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.65@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:38:05.642889 5390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:05.629381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:34439 (local address 127.16.233.126:38213)
0810 22:38:05.629536 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:05.629540 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:05.630368 (+ 828us) server_negotiation.cc:407] Connection header received
0810 22:38:05.631702 (+ 1334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:05.631705 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:05.631755 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:05.631835 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:05.633293 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:05.633849 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:05.635163 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:05.635421 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:05.635519 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:05.638933 (+ 3414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:05.638938 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:05.638940 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:38:05.638976 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:05.640462 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:05.641121 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:05.641125 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:05.641127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:05.641215 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:05.641594 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:05.641598 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:05.641599 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:05.641843 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:05.641963 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:05.642559 (+ 596us) server_negotiation.cc:299] Negotiation successful
0810 22:38:05.642700 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 22:38:05.643766 5557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:05.629675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:38213 (local address 127.16.233.65:34439)
0810 22:38:05.630172 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:05.630217 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 22:38:05.631494 (+ 1277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:38:05.632073 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:38:05.632080 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:38:05.632462 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:38:05.633119 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:05.633137 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:05.634000 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:05.634003 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:05.635020 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:05.635030 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:05.635628 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:05.635635 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:05.635939 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:05.636732 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:38:05.636753 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:38:05.638753 (+ 2000us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:38:05.640608 (+ 1855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:38:05.640615 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:38:05.640633 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:38:05.641011 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:38:05.641323 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:38:05.641326 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:38:05.641332 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:38:05.641487 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:38:05.641959 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:38:05.641965 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:38:05.642291 (+ 326us) client_negotiation.cc:769] Sending connection context
0810 22:38:05.642502 (+ 211us) client_negotiation.cc:240] Negotiation successful
0810 22:38:05.642861 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":120,"threads_started":1}
I20260810 22:38:05.645167 5553 heartbeater.cc:347] Connected to a master server at 127.16.233.126:38213
I20260810 22:38:05.645453 5553 heartbeater.cc:464] Registering TS with master...
I20260810 22:38:05.645988 5553 heartbeater.cc:511] Master 127.16.233.126:38213 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.16.233.66@KRBTEST.COM; defaulting to no policy
I20260810 22:38:05.647748 5345 ts_manager.cc:194] Registered new tserver with Master: 178b06fdf44849fe9daac479a7651835 (127.16.233.65:34743)
I20260810 22:38:05.649071 5345 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.65@KRBTEST.COM'} at 127.16.233.65:34439
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab.
Entry for principal kudu/127.16.233.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab.
I20260810 22:38:05.672060 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.66:0
--local_ip_for_outbound_sockets=127.16.233.66
--webserver_interface=127.16.233.66
--webserver_port=0
--tserver_master_addrs=127.16.233.126:38213
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab
--principal=kudu/127.16.233.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:37931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:38:05.796494 5562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:38:05.796800 5562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:38:05.796955 5562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:38:05.801354 5562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:38:05.801433 5562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:38:05.801496 5562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.66
I20260810 22:38:05.807278 5562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:37931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab
--principal=kudu/127.16.233.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.233.66
--webserver_port=0
--tserver_master_addrs=127.16.233.126:38213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.66
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:38:05.808590 5562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:38:05.809751 5562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:38:05.820333 5568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:05.820287 5567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:05.820279 5570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:38:05.820744 5562 server_base.cc:1034] running on GCE node
I20260810 22:38:05.821277 5562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:38:05.822007 5562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:38:05.823182 5562 hybrid_clock.cc:648] HybridClock initialized: now 1786401485823161 us; error 32 us; skew 500 ppm
Aug 10 22:38:05 dist-test-slave-77v9 krb5kdc[5294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401485, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:38:05.826342 5562 init.cc:377] Logged in from keytab as kudu/127.16.233.66@KRBTEST.COM (short username kudu)
I20260810 22:38:05.827441 5562 webserver.cc:460] Webserver started at http://127.16.233.66:35251/ using document root <none> and password file <none>
I20260810 22:38:05.828020 5562 fs_manager.cc:359] Metadata directory not provided
I20260810 22:38:05.828083 5562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:38:05.828264 5562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:38:05.830626 5562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/minicluster-data/ts-1/data/instance:
uuid: "742b05d83e654390a607a842148c7463"
format_stamp: "Formatted at 2026-08-10 22:38:05 on dist-test-slave-77v9"
server_key: "4dda12c002a372ef11d19841b4c931fa"
server_key_iv: "271de8aaa1b4fd6c1a7463e098e0e523"
server_key_version: "encryptionkey@0"
I20260810 22:38:05.831194 5562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/minicluster-data/ts-1/wal/instance:
uuid: "742b05d83e654390a607a842148c7463"
format_stamp: "Formatted at 2026-08-10 22:38:05 on dist-test-slave-77v9"
server_key: "4dda12c002a372ef11d19841b4c931fa"
server_key_iv: "271de8aaa1b4fd6c1a7463e098e0e523"
server_key_version: "encryptionkey@0"
I20260810 22:38:05.835276 5562 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 22:38:05.838341 5577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:05.839871 5562 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:38:05.840030 5562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/minicluster-data/ts-1/wal
uuid: "742b05d83e654390a607a842148c7463"
format_stamp: "Formatted at 2026-08-10 22:38:05 on dist-test-slave-77v9"
server_key: "4dda12c002a372ef11d19841b4c931fa"
server_key_iv: "271de8aaa1b4fd6c1a7463e098e0e523"
server_key_version: "encryptionkey@0"
I20260810 22:38:05.840178 5562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:38:05.861088 5562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:38:05.864712 5562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:38:05.865020 5562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:38:05.865681 5562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:38:05.866622 5562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:38:05.866700 5562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:05.866771 5562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:38:05.866817 5562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:05.876755 5562 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.66:38719
I20260810 22:38:05.876775 5700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.66:38719 every 8 connection(s)
I20260810 22:38:05.877753 5562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
I20260810 22:38:05.881502 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 5562
I20260810 22:38:05.881654 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/minicluster-data/ts-1/wal/instance
I20260810 22:38:05.881971 17317 external_mini_cluster.cc:1428] Setting key 67f038ea288958c53bfbb26b9ee31bd0
Aug 10 22:38:05 dist-test-slave-77v9 krb5kdc[5294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401485, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.66@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.233.67@KRBTEST.COM; defaulting to no policy
I20260810 22:38:05.895275 5390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:05.879724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:60987 (local address 127.16.233.126:38213)
0810 22:38:05.879872 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:05.879877 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:05.880789 (+ 912us) server_negotiation.cc:407] Connection header received
0810 22:38:05.882576 (+ 1787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:05.882580 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:05.882640 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:05.882743 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:05.884314 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:05.885207 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:05.886942 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:05.887175 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:05.887262 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:05.890540 (+ 3278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:05.890544 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:05.890546 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:38:05.890579 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:05.892426 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:05.893245 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:05.893249 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:05.893250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:05.893357 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:05.893863 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:05.893867 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:05.893868 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:05.894095 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:05.894189 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:05.894988 (+ 799us) server_negotiation.cc:299] Negotiation successful
0810 22:38:05.895121 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 22:38:05.896291 5704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:05.880065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:38213 (local address 127.16.233.66:60987)
0810 22:38:05.880627 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:05.880681 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 22:38:05.882275 (+ 1594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:38:05.883047 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:38:05.883057 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:38:05.883618 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:38:05.884156 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:05.884170 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:05.885336 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:05.885338 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:05.886783 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:05.886798 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:05.887348 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:05.887351 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:05.887499 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:05.888175 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:38:05.888193 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:38:05.890263 (+ 2070us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:38:05.892585 (+ 2322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:38:05.892597 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:38:05.892619 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:38:05.893117 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:38:05.893483 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:38:05.893487 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:38:05.893489 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:38:05.893708 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:38:05.894222 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:38:05.894230 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:38:05.894638 (+ 408us) client_negotiation.cc:769] Sending connection context
0810 22:38:05.894894 (+ 256us) client_negotiation.cc:240] Negotiation successful
0810 22:38:05.895240 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":147,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.67@KRBTEST.COM" created.
I20260810 22:38:05.898031 5701 heartbeater.cc:347] Connected to a master server at 127.16.233.126:38213
I20260810 22:38:05.898329 5701 heartbeater.cc:464] Registering TS with master...
I20260810 22:38:05.898922 5701 heartbeater.cc:511] Master 127.16.233.126:38213 requested a full tablet report, sending...
I20260810 22:38:05.900158 5345 ts_manager.cc:194] Registered new tserver with Master: 742b05d83e654390a607a842148c7463 (127.16.233.66:38719)
I20260810 22:38:05.901116 5345 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.66@KRBTEST.COM'} at 127.16.233.66:60987
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab.
Entry for principal kudu/127.16.233.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab.
I20260810 22:38:05.917165 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.67:0
--local_ip_for_outbound_sockets=127.16.233.67
--webserver_interface=127.16.233.67
--webserver_port=0
--tserver_master_addrs=127.16.233.126:38213
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab
--principal=kudu/127.16.233.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:37931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:38:06.034915 5708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:38:06.035264 5708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:38:06.035311 5708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:38:06.039857 5708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:38:06.039956 5708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:38:06.040022 5708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.67
I20260810 22:38:06.045961 5708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:37931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab
--principal=kudu/127.16.233.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.233.67
--webserver_port=0
--tserver_master_addrs=127.16.233.126:38213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.67
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:38:06.047116 5708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:38:06.048123 5708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:38:06.059412 5715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:06.059429 5714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:06.059429 5717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:38:06.059552 5708 server_base.cc:1034] running on GCE node
I20260810 22:38:06.060469 5708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:38:06.061316 5708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:38:06.062511 5708 hybrid_clock.cc:648] HybridClock initialized: now 1786401486062462 us; error 65 us; skew 500 ppm
Aug 10 22:38:06 dist-test-slave-77v9 krb5kdc[5294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401486, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:38:06.065976 5708 init.cc:377] Logged in from keytab as kudu/127.16.233.67@KRBTEST.COM (short username kudu)
I20260810 22:38:06.067037 5708 webserver.cc:460] Webserver started at http://127.16.233.67:35567/ using document root <none> and password file <none>
I20260810 22:38:06.067698 5708 fs_manager.cc:359] Metadata directory not provided
I20260810 22:38:06.067770 5708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:38:06.068006 5708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:38:06.070362 5708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/minicluster-data/ts-2/data/instance:
uuid: "4bdb1f6583a9435bad1e518f9b2ed52a"
format_stamp: "Formatted at 2026-08-10 22:38:06 on dist-test-slave-77v9"
server_key: "db6d36ee5456a3aa21526ea063ffab7c"
server_key_iv: "8dd09f2a4935173531fddaeef1b4af9c"
server_key_version: "encryptionkey@0"
I20260810 22:38:06.071004 5708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/minicluster-data/ts-2/wal/instance:
uuid: "4bdb1f6583a9435bad1e518f9b2ed52a"
format_stamp: "Formatted at 2026-08-10 22:38:06 on dist-test-slave-77v9"
server_key: "db6d36ee5456a3aa21526ea063ffab7c"
server_key_iv: "8dd09f2a4935173531fddaeef1b4af9c"
server_key_version: "encryptionkey@0"
I20260810 22:38:06.075459 5708 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 22:38:06.078507 5725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:06.080399 5708 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 22:38:06.080572 5708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/minicluster-data/ts-2/wal
uuid: "4bdb1f6583a9435bad1e518f9b2ed52a"
format_stamp: "Formatted at 2026-08-10 22:38:06 on dist-test-slave-77v9"
server_key: "db6d36ee5456a3aa21526ea063ffab7c"
server_key_iv: "8dd09f2a4935173531fddaeef1b4af9c"
server_key_version: "encryptionkey@0"
I20260810 22:38:06.080713 5708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:38:06.094340 5708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:38:06.098409 5708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:38:06.098664 5708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:38:06.099385 5708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:38:06.100337 5708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:38:06.100412 5708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:06.100492 5708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:38:06.100536 5708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:06.111047 5708 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.67:36261
I20260810 22:38:06.111064 5844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.67:36261 every 8 connection(s)
I20260810 22:38:06.112059 5708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
I20260810 22:38:06.115705 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 5708
I20260810 22:38:06.115926 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/minicluster-data/ts-2/wal/instance
I20260810 22:38:06.116264 17317 external_mini_cluster.cc:1428] Setting key f1471cc47e7c89800b78448a49d58156
Aug 10 22:38:06 dist-test-slave-77v9 krb5kdc[5294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401486, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.67@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:38:06.127460 5390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:06.114193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:38801 (local address 127.16.233.126:38213)
0810 22:38:06.114351 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:06.114355 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:06.115289 (+ 934us) server_negotiation.cc:407] Connection header received
0810 22:38:06.116973 (+ 1684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:06.116977 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:06.117040 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:06.117131 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:06.118368 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:06.118928 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:06.120160 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:06.120398 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:06.120521 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:06.123487 (+ 2966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:06.123490 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:06.123492 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:38:06.123518 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:06.125146 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:06.125725 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:06.125729 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:06.125731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:06.125820 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:06.126190 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:06.126195 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:06.126196 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:06.126416 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:06.126550 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:06.127107 (+ 557us) server_negotiation.cc:299] Negotiation successful
0810 22:38:06.127264 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 22:38:06.128326 5847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:06.114568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:38213 (local address 127.16.233.67:38801)
0810 22:38:06.115120 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:06.115164 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 22:38:06.116693 (+ 1529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:38:06.117324 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:38:06.117331 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:38:06.117700 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:38:06.118201 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:06.118215 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:06.119066 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:06.119070 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:06.120017 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:06.120027 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:06.120570 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:06.120574 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:06.120757 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:06.121482 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:38:06.121499 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:38:06.123261 (+ 1762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:38:06.125283 (+ 2022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:38:06.125291 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:38:06.125310 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:38:06.125607 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:38:06.125924 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:38:06.125927 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:38:06.125929 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:38:06.126085 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:38:06.126552 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:38:06.126561 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:38:06.126837 (+ 276us) client_negotiation.cc:769] Sending connection context
0810 22:38:06.127051 (+ 214us) client_negotiation.cc:240] Negotiation successful
0810 22:38:06.127416 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":128,"threads_started":1}
I20260810 22:38:06.129920 5845 heartbeater.cc:347] Connected to a master server at 127.16.233.126:38213
I20260810 22:38:06.130206 5845 heartbeater.cc:464] Registering TS with master...
I20260810 22:38:06.130872 5845 heartbeater.cc:511] Master 127.16.233.126:38213 requested a full tablet report, sending...
I20260810 22:38:06.131934 5344 ts_manager.cc:194] Registered new tserver with Master: 4bdb1f6583a9435bad1e518f9b2ed52a (127.16.233.67:36261)
I20260810 22:38:06.132563 5344 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.67@KRBTEST.COM'} at 127.16.233.67:38801
I20260810 22:38:06.143172 17317 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 22:38:06.154541 5390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:06.145899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53198 (local address 127.16.233.126:38213)
0810 22:38:06.146069 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:06.146074 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:06.146270 (+ 196us) server_negotiation.cc:407] Connection header received
0810 22:38:06.146371 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:06.146373 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:06.146418 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:06.146500 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:06.147311 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:06.147884 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:06.148722 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:06.149050 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:06.149141 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:06.150815 (+ 1674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:06.150818 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:06.150820 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:38:06.150846 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:06.152336 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:06.152939 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:06.152943 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:06.152944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:06.153026 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:06.153419 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:06.153422 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:06.153423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:06.153672 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:06.153831 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:06.154174 (+ 343us) server_negotiation.cc:299] Negotiation successful
0810 22:38:06.154337 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 22:38:06.158241 5344 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53198:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 22:38:06.161834 5344 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:38:06.178145 5858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:06.171748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34628 (local address 127.16.233.65:34743)
0810 22:38:06.173001 (+ 1253us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:06.173009 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:06.173033 (+ 24us) server_negotiation.cc:407] Connection header received
0810 22:38:06.173111 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:06.173118 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:06.173314 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:06.173469 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:38:06.173781 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:06.175001 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:06.176727 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:06.177397 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:06.177525 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:06.177692 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:06.177872 (+ 180us) server_negotiation.cc:299] Negotiation successful
0810 22:38:06.177975 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1132,"thread_start_us":98,"threads_started":1}
I20260810 22:38:06.178135 5856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:06.171577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.65:34743 (local address 127.0.0.1:34628)
0810 22:38:06.172461 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:06.172518 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 22:38:06.172670 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:38:06.173474 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:38:06.173480 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:38:06.173502 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:38:06.173654 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:06.173661 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:06.175176 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:06.175181 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:06.176593 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:06.176602 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:06.177535 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:06.177538 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:06.177649 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:06.177687 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 22:38:06.177835 (+ 148us) client_negotiation.cc:240] Negotiation successful
0810 22:38:06.177967 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":713,"mutex_wait_us":34,"spinlock_wait_cycles":6144,"thread_start_us":90,"threads_started":1}
I20260810 22:38:06.178521 5863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:06.171882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.67:36261 (local address 127.0.0.1:59336)
0810 22:38:06.172502 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:06.172530 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0810 22:38:06.172669 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:38:06.173637 (+ 968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:38:06.173641 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:38:06.173663 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:38:06.173825 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:06.173832 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:06.175397 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:06.175401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:06.176686 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:06.176699 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:06.178097 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:06.178101 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:06.178188 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:06.178211 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 22:38:06.178300 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 22:38:06.178369 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"mutex_wait_us":59,"spinlock_wait_cycles":6784,"thread_start_us":114,"threads_started":1}
I20260810 22:38:06.179044 5864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:06.172300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59336 (local address 127.16.233.67:36261)
0810 22:38:06.173064 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:06.173076 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:06.173096 (+ 20us) server_negotiation.cc:407] Connection header received
0810 22:38:06.173159 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:06.173163 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:06.173330 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:06.173480 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:38:06.173956 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:06.175051 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:06.176866 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:06.177757 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:06.177870 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:06.178030 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:06.178789 (+ 759us) server_negotiation.cc:299] Negotiation successful
0810 22:38:06.178898 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":151,"threads_started":1}
I20260810 22:38:06.179915 5860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:06.171577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.66:38719 (local address 127.0.0.1:34432)
0810 22:38:06.172698 (+ 1121us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:06.172713 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:38:06.172793 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:38:06.173637 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:38:06.173640 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:38:06.173668 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:38:06.173825 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:06.173832 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:06.176538 (+ 2706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:06.176541 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:06.177840 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:06.177850 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:06.179411 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:06.179415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:06.179501 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:06.179534 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 22:38:06.179608 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 22:38:06.179776 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1065,"mutex_wait_us":620,"spinlock_wait_cycles":6144,"thread_start_us":108,"threads_started":1}
I20260810 22:38:06.180474 5862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:06.171901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34432 (local address 127.16.233.66:38719)
0810 22:38:06.173026 (+ 1125us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:06.173038 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:06.173060 (+ 22us) server_negotiation.cc:407] Connection header received
0810 22:38:06.173139 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:06.173144 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:06.173330 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:06.173480 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:38:06.175188 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:06.176374 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:06.177979 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:06.179082 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:06.179204 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:06.179342 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:06.180216 (+ 874us) server_negotiation.cc:299] Negotiation successful
0810 22:38:06.180330 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1023,"thread_start_us":119,"threads_started":1}
I20260810 22:38:06.181674 5484 tablet_service.cc:1467] Processing CreateTablet for tablet e98fdf20af334b969d0d233f5f6081d2 (DEFAULT_TABLE table=test-table [id=ac349cfd25b343a5993e43e11459e826]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:38:06.181802 5774 tablet_service.cc:1467] Processing CreateTablet for tablet e98fdf20af334b969d0d233f5f6081d2 (DEFAULT_TABLE table=test-table [id=ac349cfd25b343a5993e43e11459e826]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:38:06.182703 5484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e98fdf20af334b969d0d233f5f6081d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:38:06.182798 5774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e98fdf20af334b969d0d233f5f6081d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:38:06.185985 5630 tablet_service.cc:1467] Processing CreateTablet for tablet e98fdf20af334b969d0d233f5f6081d2 (DEFAULT_TABLE table=test-table [id=ac349cfd25b343a5993e43e11459e826]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:38:06.187337 5630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e98fdf20af334b969d0d233f5f6081d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:38:06.191380 5865 tablet_bootstrap.cc:492] T e98fdf20af334b969d0d233f5f6081d2 P 4bdb1f6583a9435bad1e518f9b2ed52a: Bootstrap starting.
I20260810 22:38:06.192848 5866 tablet_bootstrap.cc:492] T e98fdf20af334b969d0d233f5f6081d2 P 178b06fdf44849fe9daac479a7651835: Bootstrap starting.
I20260810 22:38:06.193940 5865 tablet_bootstrap.cc:654] T e98fdf20af334b969d0d233f5f6081d2 P 4bdb1f6583a9435bad1e518f9b2ed52a: Neither blocks nor log segments found. Creating new log.
I20260810 22:38:06.195052 5866 tablet_bootstrap.cc:654] T e98fdf20af334b969d0d233f5f6081d2 P 178b06fdf44849fe9daac479a7651835: Neither blocks nor log segments found. Creating new log.
I20260810 22:38:06.195551 5865 log.cc:824] T e98fdf20af334b969d0d233f5f6081d2 P 4bdb1f6583a9435bad1e518f9b2ed52a: Log is configured to *not* fsync() on all Append() calls
I20260810 22:38:06.195966 5866 log.cc:824] T e98fdf20af334b969d0d233f5f6081d2 P 178b06fdf44849fe9daac479a7651835: Log is configured to *not* fsync() on all Append() calls
I20260810 22:38:06.196430 5867 tablet_bootstrap.cc:492] T e98fdf20af334b969d0d233f5f6081d2 P 742b05d83e654390a607a842148c7463: Bootstrap starting.
I20260810 22:38:06.197835 5866 tablet_bootstrap.cc:492] T e98fdf20af334b969d0d233f5f6081d2 P 178b06fdf44849fe9daac479a7651835: No bootstrap required, opened a new log
I20260810 22:38:06.198012 5865 tablet_bootstrap.cc:492] T e98fdf20af334b969d0d233f5f6081d2 P 4bdb1f6583a9435bad1e518f9b2ed52a: No bootstrap required, opened a new log
I20260810 22:38:06.198129 5866 ts_tablet_manager.cc:1397] T e98fdf20af334b969d0d233f5f6081d2 P 178b06fdf44849fe9daac479a7651835: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 22:38:06.198221 5865 ts_tablet_manager.cc:1397] T e98fdf20af334b969d0d233f5f6081d2 P 4bdb1f6583a9435bad1e518f9b2ed52a: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 22:38:06.198915 5867 tablet_bootstrap.cc:654] T e98fdf20af334b969d0d233f5f6081d2 P 742b05d83e654390a607a842148c7463: Neither blocks nor log segments found. Creating new log.
I20260810 22:38:06.200405 5867 log.cc:824] T e98fdf20af334b969d0d233f5f6081d2 P 742b05d83e654390a607a842148c7463: Log is configured to *not* fsync() on all Append() calls
I20260810 22:38:06.202014 5866 raft_consensus.cc:348] T e98fdf20af334b969d0d233f5f6081d2 P 178b06fdf44849fe9daac479a7651835 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "178b06fdf44849fe9daac479a7651835" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 34743 } } peers { permanent_uuid: "742b05d83e654390a607a842148c7463" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 38719 } } peers { permanent_uuid: "4bdb1f6583a9435bad1e518f9b2ed52a" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 36261 } }
I20260810 22:38:06.202467 5866 raft_consensus.cc:374] T e98fdf20af334b969d0d233f5f6081d2 P 178b06fdf44849fe9daac479a7651835 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:38:06.202584 5866 raft_consensus.cc:729] T e98fdf20af334b969d0d233f5f6081d2 P 178b06fdf44849fe9daac479a7651835 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 178b06fdf44849fe9daac479a7651835, State: Initialized, Role: FOLLOWER
I20260810 22:38:06.202869 5867 tablet_bootstrap.cc:492] T e98fdf20af334b969d0d233f5f6081d2 P 742b05d83e654390a607a842148c7463: No bootstrap required, opened a new log
I20260810 22:38:06.202972 5865 raft_consensus.cc:348] T e98fdf20af334b969d0d233f5f6081d2 P 4bdb1f6583a9435bad1e518f9b2ed52a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "178b06fdf44849fe9daac479a7651835" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 34743 } } peers { permanent_uuid: "742b05d83e654390a607a842148c7463" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 38719 } } peers { permanent_uuid: "4bdb1f6583a9435bad1e518f9b2ed52a" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 36261 } }
I20260810 22:38:06.203195 5867 ts_tablet_manager.cc:1397] T e98fdf20af334b969d0d233f5f6081d2 P 742b05d83e654390a607a842148c7463: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260810 22:38:06.203150 5866 consensus_queue.cc:260] T e98fdf20af334b969d0d233f5f6081d2 P 178b06fdf44849fe9daac479a7651835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "178b06fdf44849fe9daac479a7651835" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 34743 } } peers { permanent_uuid: "742b05d83e654390a607a842148c7463" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 38719 } } peers { permanent_uuid: "4bdb1f6583a9435bad1e518f9b2ed52a" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 36261 } }
I20260810 22:38:06.203258 5865 raft_consensus.cc:374] T e98fdf20af334b969d0d233f5f6081d2 P 4bdb1f6583a9435bad1e518f9b2ed52a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:38:06.203351 5865 raft_consensus.cc:729] T e98fdf20af334b969d0d233f5f6081d2 P 4bdb1f6583a9435bad1e518f9b2ed52a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bdb1f6583a9435bad1e518f9b2ed52a, State: Initialized, Role: FOLLOWER
I20260810 22:38:06.203858 5553 heartbeater.cc:503] Master 127.16.233.126:38213 was elected leader, sending a full tablet report...
I20260810 22:38:06.203886 5865 consensus_queue.cc:260] T e98fdf20af334b969d0d233f5f6081d2 P 4bdb1f6583a9435bad1e518f9b2ed52a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "178b06fdf44849fe9daac479a7651835" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 34743 } } peers { permanent_uuid: "742b05d83e654390a607a842148c7463" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 38719 } } peers { permanent_uuid: "4bdb1f6583a9435bad1e518f9b2ed52a" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 36261 } }
I20260810 22:38:06.204602 5845 heartbeater.cc:503] Master 127.16.233.126:38213 was elected leader, sending a full tablet report...
I20260810 22:38:06.205073 5865 ts_tablet_manager.cc:1428] T e98fdf20af334b969d0d233f5f6081d2 P 4bdb1f6583a9435bad1e518f9b2ed52a: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260810 22:38:06.206884 5866 ts_tablet_manager.cc:1428] T e98fdf20af334b969d0d233f5f6081d2 P 178b06fdf44849fe9daac479a7651835: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260810 22:38:06.207739 5867 raft_consensus.cc:348] T e98fdf20af334b969d0d233f5f6081d2 P 742b05d83e654390a607a842148c7463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "178b06fdf44849fe9daac479a7651835" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 34743 } } peers { permanent_uuid: "742b05d83e654390a607a842148c7463" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 38719 } } peers { permanent_uuid: "4bdb1f6583a9435bad1e518f9b2ed52a" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 36261 } }
I20260810 22:38:06.208086 5867 raft_consensus.cc:374] T e98fdf20af334b969d0d233f5f6081d2 P 742b05d83e654390a607a842148c7463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:38:06.208211 5867 raft_consensus.cc:729] T e98fdf20af334b969d0d233f5f6081d2 P 742b05d83e654390a607a842148c7463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 742b05d83e654390a607a842148c7463, State: Initialized, Role: FOLLOWER
I20260810 22:38:06.208778 5867 consensus_queue.cc:260] T e98fdf20af334b969d0d233f5f6081d2 P 742b05d83e654390a607a842148c7463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "178b06fdf44849fe9daac479a7651835" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 34743 } } peers { permanent_uuid: "742b05d83e654390a607a842148c7463" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 38719 } } peers { permanent_uuid: "4bdb1f6583a9435bad1e518f9b2ed52a" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 36261 } }
I20260810 22:38:06.209684 5701 heartbeater.cc:503] Master 127.16.233.126:38213 was elected leader, sending a full tablet report...
I20260810 22:38:06.210048 5867 ts_tablet_manager.cc:1428] T e98fdf20af334b969d0d233f5f6081d2 P 742b05d83e654390a607a842148c7463: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260810 22:38:06.363147 5846 tablet.cc:2369] T e98fdf20af334b969d0d233f5f6081d2 P 4bdb1f6583a9435bad1e518f9b2ed52a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 22:38:06.379099 5555 tablet.cc:2369] T e98fdf20af334b969d0d233f5f6081d2 P 178b06fdf44849fe9daac479a7651835: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 22:38:06.379132 5703 tablet.cc:2369] T e98fdf20af334b969d0d233f5f6081d2 P 742b05d83e654390a607a842148c7463: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:38:06.471612 5872 raft_consensus.cc:482] T e98fdf20af334b969d0d233f5f6081d2 P 4bdb1f6583a9435bad1e518f9b2ed52a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:38:06.472011 5872 raft_consensus.cc:504] T e98fdf20af334b969d0d233f5f6081d2 P 4bdb1f6583a9435bad1e518f9b2ed52a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "178b06fdf44849fe9daac479a7651835" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 34743 } } peers { permanent_uuid: "742b05d83e654390a607a842148c7463" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 38719 } } peers { permanent_uuid: "4bdb1f6583a9435bad1e518f9b2ed52a" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 36261 } }
I20260810 22:38:06.473719 5872 leader_election.cc:288] T e98fdf20af334b969d0d233f5f6081d2 P 4bdb1f6583a9435bad1e518f9b2ed52a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 178b06fdf44849fe9daac479a7651835 (127.16.233.65:34743), 742b05d83e654390a607a842148c7463 (127.16.233.66:38719)
I20260810 22:38:06.477571 5847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:06.473976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.65:34743 (local address 127.16.233.67:43289)
0810 22:38:06.474143 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:06.474160 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 22:38:06.474261 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:38:06.474593 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:38:06.474596 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:38:06.474616 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:38:06.474745 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:06.474750 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:06.475524 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:06.475529 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:06.476315 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:06.476324 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:06.477116 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:06.477120 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:06.477236 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:06.477254 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 22:38:06.477353 (+ 99us) client_negotiation.cc:240] Negotiation successful
0810 22:38:06.477430 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260810 22:38:06.477571 5858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:06.474045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:43289 (local address 127.16.233.65:34743)
0810 22:38:06.474179 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:06.474183 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:06.474201 (+ 18us) server_negotiation.cc:407] Connection header received
0810 22:38:06.474406 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:06.474411 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:06.474471 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:06.474578 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:38:06.474886 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:06.475405 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:06.476466 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:06.476982 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:06.477119 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:06.477177 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:06.477364 (+ 187us) server_negotiation.cc:299] Negotiation successful
0810 22:38:06.477445 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 22:38:06.477960 5862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:06.474205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:58587 (local address 127.16.233.66:38719)
0810 22:38:06.474395 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:06.474400 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:06.474758 (+ 358us) server_negotiation.cc:407] Connection header received
0810 22:38:06.474947 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:06.474950 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:06.475002 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:06.475088 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:38:06.475375 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:06.475862 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:06.476935 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:06.477342 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:06.477478 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:06.477539 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:06.477764 (+ 225us) server_negotiation.cc:299] Negotiation successful
0810 22:38:06.477845 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 22:38:06.478322 5874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:06.473988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.66:38719 (local address 127.16.233.67:58587)
0810 22:38:06.474653 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:06.474665 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 22:38:06.474830 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:38:06.475104 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:38:06.475107 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:38:06.475122 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:38:06.475257 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:06.475263 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:06.475976 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:06.475980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:06.476754 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:06.476762 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:06.477469 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:06.477472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:06.477571 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:06.477592 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 22:38:06.478118 (+ 526us) client_negotiation.cc:240] Negotiation successful
0810 22:38:06.478198 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"mutex_wait_us":286,"thread_start_us":159,"threads_started":1}
I20260810 22:38:06.478309 5506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e98fdf20af334b969d0d233f5f6081d2" candidate_uuid: "4bdb1f6583a9435bad1e518f9b2ed52a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "178b06fdf44849fe9daac479a7651835" is_pre_election: true
I20260810 22:38:06.478639 5506 raft_consensus.cc:2454] T e98fdf20af334b969d0d233f5f6081d2 P 178b06fdf44849fe9daac479a7651835 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bdb1f6583a9435bad1e518f9b2ed52a in term 0.
I20260810 22:38:06.479015 5651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e98fdf20af334b969d0d233f5f6081d2" candidate_uuid: "4bdb1f6583a9435bad1e518f9b2ed52a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "742b05d83e654390a607a842148c7463" is_pre_election: true
I20260810 22:38:06.479198 5726 leader_election.cc:302] T e98fdf20af334b969d0d233f5f6081d2 P 4bdb1f6583a9435bad1e518f9b2ed52a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 178b06fdf44849fe9daac479a7651835, 4bdb1f6583a9435bad1e518f9b2ed52a; no voters:
I20260810 22:38:06.479238 5651 raft_consensus.cc:2454] T e98fdf20af334b969d0d233f5f6081d2 P 742b05d83e654390a607a842148c7463 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bdb1f6583a9435bad1e518f9b2ed52a in term 0.
I20260810 22:38:06.479552 5872 raft_consensus.cc:2781] T e98fdf20af334b969d0d233f5f6081d2 P 4bdb1f6583a9435bad1e518f9b2ed52a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:38:06.479665 5872 raft_consensus.cc:482] T e98fdf20af334b969d0d233f5f6081d2 P 4bdb1f6583a9435bad1e518f9b2ed52a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:38:06.479746 5872 raft_consensus.cc:3037] T e98fdf20af334b969d0d233f5f6081d2 P 4bdb1f6583a9435bad1e518f9b2ed52a [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:38:06.480963 5872 raft_consensus.cc:504] T e98fdf20af334b969d0d233f5f6081d2 P 4bdb1f6583a9435bad1e518f9b2ed52a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "178b06fdf44849fe9daac479a7651835" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 34743 } } peers { permanent_uuid: "742b05d83e654390a607a842148c7463" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 38719 } } peers { permanent_uuid: "4bdb1f6583a9435bad1e518f9b2ed52a" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 36261 } }
I20260810 22:38:06.481352 5872 leader_election.cc:288] T e98fdf20af334b969d0d233f5f6081d2 P 4bdb1f6583a9435bad1e518f9b2ed52a [CANDIDATE]: Term 1 election: Requested vote from peers 178b06fdf44849fe9daac479a7651835 (127.16.233.65:34743), 742b05d83e654390a607a842148c7463 (127.16.233.66:38719)
I20260810 22:38:06.481714 5506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e98fdf20af334b969d0d233f5f6081d2" candidate_uuid: "4bdb1f6583a9435bad1e518f9b2ed52a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "178b06fdf44849fe9daac479a7651835"
I20260810 22:38:06.481730 5651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e98fdf20af334b969d0d233f5f6081d2" candidate_uuid: "4bdb1f6583a9435bad1e518f9b2ed52a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "742b05d83e654390a607a842148c7463"
I20260810 22:38:06.481889 5651 raft_consensus.cc:3037] T e98fdf20af334b969d0d233f5f6081d2 P 742b05d83e654390a607a842148c7463 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:38:06.481922 5506 raft_consensus.cc:3037] T e98fdf20af334b969d0d233f5f6081d2 P 178b06fdf44849fe9daac479a7651835 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:38:06.482967 5651 raft_consensus.cc:2454] T e98fdf20af334b969d0d233f5f6081d2 P 742b05d83e654390a607a842148c7463 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bdb1f6583a9435bad1e518f9b2ed52a in term 1.
I20260810 22:38:06.483063 5506 raft_consensus.cc:2454] T e98fdf20af334b969d0d233f5f6081d2 P 178b06fdf44849fe9daac479a7651835 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bdb1f6583a9435bad1e518f9b2ed52a in term 1.
I20260810 22:38:06.483330 5727 leader_election.cc:302] T e98fdf20af334b969d0d233f5f6081d2 P 4bdb1f6583a9435bad1e518f9b2ed52a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bdb1f6583a9435bad1e518f9b2ed52a, 742b05d83e654390a607a842148c7463; no voters:
I20260810 22:38:06.483517 5872 raft_consensus.cc:2781] T e98fdf20af334b969d0d233f5f6081d2 P 4bdb1f6583a9435bad1e518f9b2ed52a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:38:06.483759 5872 raft_consensus.cc:686] T e98fdf20af334b969d0d233f5f6081d2 P 4bdb1f6583a9435bad1e518f9b2ed52a [term 1 LEADER]: Becoming Leader. State: Replica: 4bdb1f6583a9435bad1e518f9b2ed52a, State: Running, Role: LEADER
I20260810 22:38:06.484094 5872 consensus_queue.cc:237] T e98fdf20af334b969d0d233f5f6081d2 P 4bdb1f6583a9435bad1e518f9b2ed52a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "178b06fdf44849fe9daac479a7651835" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 34743 } } peers { permanent_uuid: "742b05d83e654390a607a842148c7463" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 38719 } } peers { permanent_uuid: "4bdb1f6583a9435bad1e518f9b2ed52a" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 36261 } }
I20260810 22:38:06.487427 5345 catalog_manager.cc:5495] T e98fdf20af334b969d0d233f5f6081d2 P 4bdb1f6583a9435bad1e518f9b2ed52a reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bdb1f6583a9435bad1e518f9b2ed52a (127.16.233.67). New cstate: current_term: 1 leader_uuid: "4bdb1f6583a9435bad1e518f9b2ed52a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "178b06fdf44849fe9daac479a7651835" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 34743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "742b05d83e654390a607a842148c7463" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 38719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bdb1f6583a9435bad1e518f9b2ed52a" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 36261 } health_report { overall_health: HEALTHY } } }
I20260810 22:38:06.548260 5864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:06.544207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59346 (local address 127.16.233.67:36261)
0810 22:38:06.544405 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:06.544410 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:06.544428 (+ 18us) server_negotiation.cc:407] Connection header received
0810 22:38:06.544587 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:06.544589 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:06.544714 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:06.544875 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:38:06.545226 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:06.545748 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:06.546628 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:06.546857 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:06.546973 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:06.547317 (+ 344us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:38:06.547756 (+ 439us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:38:06.547922 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:06.548029 (+ 107us) server_negotiation.cc:299] Negotiation successful
0810 22:38:06.548071 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 22:38:06.554888 5506 raft_consensus.cc:1261] T e98fdf20af334b969d0d233f5f6081d2 P 178b06fdf44849fe9daac479a7651835 [term 1 FOLLOWER]: Refusing update from remote peer 4bdb1f6583a9435bad1e518f9b2ed52a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:38:06.554927 5651 raft_consensus.cc:1261] T e98fdf20af334b969d0d233f5f6081d2 P 742b05d83e654390a607a842148c7463 [term 1 FOLLOWER]: Refusing update from remote peer 4bdb1f6583a9435bad1e518f9b2ed52a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:38:06.555716 5875 consensus_queue.cc:1035] T e98fdf20af334b969d0d233f5f6081d2 P 4bdb1f6583a9435bad1e518f9b2ed52a [LEADER]: Connected to new peer: Peer: permanent_uuid: "178b06fdf44849fe9daac479a7651835" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 34743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:38:06.555949 5872 consensus_queue.cc:1035] T e98fdf20af334b969d0d233f5f6081d2 P 4bdb1f6583a9435bad1e518f9b2ed52a [LEADER]: Connected to new peer: Peer: permanent_uuid: "742b05d83e654390a607a842148c7463" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 38719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:38:06.564525 5880 mvcc.cc:205] Tried to move back new op lower bound from 7317100488919568384 to 7317100488641740800. Current Snapshot: MvccSnapshot[applied={T|T < 7317100488919568384}]
I20260810 22:38:06.575160 5345 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:38:06.575364 5345 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:38:06.578424 5345 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5b8905fabd214d7b8b7294fe3c6f2fea: Sending DeleteTablet for 3 replicas of tablet e98fdf20af334b969d0d233f5f6081d2
I20260810 22:38:06.579487 5484 tablet_service.cc:1514] Processing DeleteTablet for tablet e98fdf20af334b969d0d233f5f6081d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:38:06 UTC) from {username='kudu', principal='kudu/127.16.233.126@KRBTEST.COM'} at 127.0.0.1:34628
I20260810 22:38:06.579797 5774 tablet_service.cc:1514] Processing DeleteTablet for tablet e98fdf20af334b969d0d233f5f6081d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:38:06 UTC) from {username='kudu', principal='kudu/127.16.233.126@KRBTEST.COM'} at 127.0.0.1:59336
I20260810 22:38:06.580156 5890 tablet_replica.cc:330] T e98fdf20af334b969d0d233f5f6081d2 P 178b06fdf44849fe9daac479a7651835: stopping tablet replica
I20260810 22:38:06.580648 5890 raft_consensus.cc:2229] T e98fdf20af334b969d0d233f5f6081d2 P 178b06fdf44849fe9daac479a7651835 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:38:06.580650 5630 tablet_service.cc:1514] Processing DeleteTablet for tablet e98fdf20af334b969d0d233f5f6081d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:38:06 UTC) from {username='kudu', principal='kudu/127.16.233.126@KRBTEST.COM'} at 127.0.0.1:34432
I20260810 22:38:06.581120 5890 raft_consensus.cc:2258] T e98fdf20af334b969d0d233f5f6081d2 P 178b06fdf44849fe9daac479a7651835 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:38:06.581537 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 5417
I20260810 22:38:06.585265 5892 tablet_replica.cc:330] T e98fdf20af334b969d0d233f5f6081d2 P 742b05d83e654390a607a842148c7463: stopping tablet replica
I20260810 22:38:06.585703 5892 raft_consensus.cc:2229] T e98fdf20af334b969d0d233f5f6081d2 P 742b05d83e654390a607a842148c7463 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:38:06.586158 5892 raft_consensus.cc:2258] T e98fdf20af334b969d0d233f5f6081d2 P 742b05d83e654390a607a842148c7463 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:38:06.587863 5892 ts_tablet_manager.cc:1905] T e98fdf20af334b969d0d233f5f6081d2 P 742b05d83e654390a607a842148c7463: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:38:06.590626 5892 ts_tablet_manager.cc:1918] T e98fdf20af334b969d0d233f5f6081d2 P 742b05d83e654390a607a842148c7463: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:38:06.590782 5892 log.cc:1196] T e98fdf20af334b969d0d233f5f6081d2 P 742b05d83e654390a607a842148c7463: Deleting WAL directory at /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/minicluster-data/ts-1/wal/wals/e98fdf20af334b969d0d233f5f6081d2
I20260810 22:38:06.591171 5892 ts_tablet_manager.cc:1939] T e98fdf20af334b969d0d233f5f6081d2 P 742b05d83e654390a607a842148c7463: Deleting consensus metadata
I20260810 22:38:06.591310 5891 tablet_replica.cc:330] T e98fdf20af334b969d0d233f5f6081d2 P 4bdb1f6583a9435bad1e518f9b2ed52a: stopping tablet replica
I20260810 22:38:06.591769 5891 raft_consensus.cc:2229] T e98fdf20af334b969d0d233f5f6081d2 P 4bdb1f6583a9435bad1e518f9b2ed52a [term 1 LEADER]: Raft consensus shutting down.
I20260810 22:38:06.592286 5891 raft_consensus.cc:2258] T e98fdf20af334b969d0d233f5f6081d2 P 4bdb1f6583a9435bad1e518f9b2ed52a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:38:06.592841 5328 catalog_manager.cc:4841] TS 742b05d83e654390a607a842148c7463 (127.16.233.66:38719): tablet e98fdf20af334b969d0d233f5f6081d2 (table test-table [id=ac349cfd25b343a5993e43e11459e826]) successfully deleted
I20260810 22:38:06.594044 5891 ts_tablet_manager.cc:1905] T e98fdf20af334b969d0d233f5f6081d2 P 4bdb1f6583a9435bad1e518f9b2ed52a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 22:38:06.594451 5327 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.233.65:34743 (error 108)
I20260810 22:38:06.595393 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 5562
I20260810 22:38:06.596019 5860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:06.595667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.65:34743 (local address 127.0.0.1:34634)
0810 22:38:06.595858 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:06.595925 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.65:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
W20260810 22:38:06.596189 5327 catalog_manager.cc:4584] TS 178b06fdf44849fe9daac479a7651835 (127.16.233.65:34743): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e98fdf20af334b969d0d233f5f6081d2: Network error: Client connection negotiation failed: client connection to 127.16.233.65:34743: connect: Connection refused (error 111)
I20260810 22:38:06.597594 5891 ts_tablet_manager.cc:1918] T e98fdf20af334b969d0d233f5f6081d2 P 4bdb1f6583a9435bad1e518f9b2ed52a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:38:06.597702 5891 log.cc:1196] T e98fdf20af334b969d0d233f5f6081d2 P 4bdb1f6583a9435bad1e518f9b2ed52a: Deleting WAL directory at /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786401215402979-17317-0/minicluster-data/ts-2/wal/wals/e98fdf20af334b969d0d233f5f6081d2
I20260810 22:38:06.598207 5891 ts_tablet_manager.cc:1939] T e98fdf20af334b969d0d233f5f6081d2 P 4bdb1f6583a9435bad1e518f9b2ed52a: Deleting consensus metadata
I20260810 22:38:06.599983 5332 catalog_manager.cc:4841] TS 4bdb1f6583a9435bad1e518f9b2ed52a (127.16.233.67:36261): tablet e98fdf20af334b969d0d233f5f6081d2 (table test-table [id=ac349cfd25b343a5993e43e11459e826]) successfully deleted
I20260810 22:38:06.606503 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 5708
I20260810 22:38:06.614800 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 5309
2026-08-10T22:38:06Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2804 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-10T22:38:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:38:06Z Disabled control of system clock
I20260810 22:38:06.651991 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786401215402979-17317-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786401215402979-17317-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786401215402979-17317-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:38543
--webserver_interface=127.16.233.126
--webserver_port=0
--builtin_ntp_servers=127.16.233.84:36783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.126:38543
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 22:38:06.780484 5897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:38:06.780793 5897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:38:06.780923 5897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:38:06.785220 5897 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:38:06.785307 5897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:38:06.785322 5897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:38:06.785339 5897 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:38:06.785356 5897 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:38:06.791003 5897 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:36783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786401215402979-17317-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786401215402979-17317-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.126:38543
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:38543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786401215402979-17317-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:38:06.792255 5897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:38:06.793794 5897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:38:06.803360 5902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:38:06.803498 5897 server_base.cc:1034] running on GCE node
W20260810 22:38:06.803360 5903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:06.803684 5905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:38:06.804387 5897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:38:06.805677 5897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:38:06.806902 5897 hybrid_clock.cc:648] HybridClock initialized: now 1786401486806876 us; error 47 us; skew 500 ppm
I20260810 22:38:06.809350 5897 webserver.cc:460] Webserver started at http://127.16.233.126:32803/ using document root <none> and password file <none>
I20260810 22:38:06.810113 5897 fs_manager.cc:359] Metadata directory not provided
I20260810 22:38:06.810191 5897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:38:06.810433 5897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:38:06.812704 5897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786401215402979-17317-0/minicluster-data/master-0/data/instance:
uuid: "e2be9adc1a284f8b92e98861accc6865"
format_stamp: "Formatted at 2026-08-10 22:38:06 on dist-test-slave-77v9"
server_key: "94f52ae7bb0d8a0c62f0e56d1ccde345"
server_key_iv: "08170ef6b4fe56f43f1fa7fb715bf597"
server_key_version: "encryptionkey@0"
I20260810 22:38:06.813409 5897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786401215402979-17317-0/minicluster-data/master-0/wal/instance:
uuid: "e2be9adc1a284f8b92e98861accc6865"
format_stamp: "Formatted at 2026-08-10 22:38:06 on dist-test-slave-77v9"
server_key: "94f52ae7bb0d8a0c62f0e56d1ccde345"
server_key_iv: "08170ef6b4fe56f43f1fa7fb715bf597"
server_key_version: "encryptionkey@0"
I20260810 22:38:06.817480 5897 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 22:38:06.820259 5911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:06.821797 5897 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:38:06.821959 5897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786401215402979-17317-0/minicluster-data/master-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786401215402979-17317-0/minicluster-data/master-0/wal
uuid: "e2be9adc1a284f8b92e98861accc6865"
format_stamp: "Formatted at 2026-08-10 22:38:06 on dist-test-slave-77v9"
server_key: "94f52ae7bb0d8a0c62f0e56d1ccde345"
server_key_iv: "08170ef6b4fe56f43f1fa7fb715bf597"
server_key_version: "encryptionkey@0"
I20260810 22:38:06.822100 5897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786401215402979-17317-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786401215402979-17317-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786401215402979-17317-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:38:06.844901 5897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:38:06.845808 5897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:38:06.846060 5897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:38:06.854616 5897 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.126:38543
I20260810 22:38:06.854647 5968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.126:38543 every 8 connection(s)
I20260810 22:38:06.855734 5897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
I20260810 22:38:06.859349 5969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:38:06.860582 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 5897
I20260810 22:38:06.860749 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786401215402979-17317-0/minicluster-data/master-0/wal/instance
I20260810 22:38:06.861181 17317 external_mini_cluster.cc:1428] Setting key bedf00cd9127a02648dacf4736e7c96f
I20260810 22:38:06.866381 5969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2be9adc1a284f8b92e98861accc6865: Bootstrap starting.
I20260810 22:38:06.868971 5969 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e2be9adc1a284f8b92e98861accc6865: Neither blocks nor log segments found. Creating new log.
I20260810 22:38:06.869864 5969 log.cc:824] T 00000000000000000000000000000000 P e2be9adc1a284f8b92e98861accc6865: Log is configured to *not* fsync() on all Append() calls
I20260810 22:38:06.871265 5972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:06.862942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41408 (local address 127.16.233.126:38543)
0810 22:38:06.863674 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:06.863694 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:06.863733 (+ 39us) server_negotiation.cc:407] Connection header received
0810 22:38:06.864416 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:06.864434 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:06.864908 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:06.865249 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:06.865697 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:06.866750 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:06.867691 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:06.868161 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:06.868349 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:06.869413 (+ 1064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:06.869418 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:06.869431 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:38:06.869847 (+ 416us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:06.869906 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:38:06.869927 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:06.870042 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:06.870161 (+ 119us) server_negotiation.cc:299] Negotiation successful
0810 22:38:06.870443 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":154,"threads_started":1}
I20260810 22:38:06.872272 5969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2be9adc1a284f8b92e98861accc6865: No bootstrap required, opened a new log
I20260810 22:38:06.876596 5969 raft_consensus.cc:348] T 00000000000000000000000000000000 P e2be9adc1a284f8b92e98861accc6865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2be9adc1a284f8b92e98861accc6865" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 38543 } }
I20260810 22:38:06.876914 5969 raft_consensus.cc:374] T 00000000000000000000000000000000 P e2be9adc1a284f8b92e98861accc6865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:38:06.876959 5969 raft_consensus.cc:729] T 00000000000000000000000000000000 P e2be9adc1a284f8b92e98861accc6865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2be9adc1a284f8b92e98861accc6865, State: Initialized, Role: FOLLOWER
I20260810 22:38:06.877404 5969 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2be9adc1a284f8b92e98861accc6865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2be9adc1a284f8b92e98861accc6865" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 38543 } }
I20260810 22:38:06.877538 5969 raft_consensus.cc:388] T 00000000000000000000000000000000 P e2be9adc1a284f8b92e98861accc6865 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:38:06.877583 5969 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2be9adc1a284f8b92e98861accc6865 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:38:06.877676 5969 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2be9adc1a284f8b92e98861accc6865 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:38:06.878939 5969 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2be9adc1a284f8b92e98861accc6865 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2be9adc1a284f8b92e98861accc6865" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 38543 } }
I20260810 22:38:06.879359 5969 leader_election.cc:302] T 00000000000000000000000000000000 P e2be9adc1a284f8b92e98861accc6865 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e2be9adc1a284f8b92e98861accc6865; no voters:
I20260810 22:38:06.879709 5969 leader_election.cc:288] T 00000000000000000000000000000000 P e2be9adc1a284f8b92e98861accc6865 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:38:06.879874 5974 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2be9adc1a284f8b92e98861accc6865 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:38:06.880142 5974 raft_consensus.cc:686] T 00000000000000000000000000000000 P e2be9adc1a284f8b92e98861accc6865 [term 1 LEADER]: Becoming Leader. State: Replica: e2be9adc1a284f8b92e98861accc6865, State: Running, Role: LEADER
I20260810 22:38:06.880466 5974 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2be9adc1a284f8b92e98861accc6865 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2be9adc1a284f8b92e98861accc6865" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 38543 } }
I20260810 22:38:06.881106 5969 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2be9adc1a284f8b92e98861accc6865 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:38:06.882401 5975 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2be9adc1a284f8b92e98861accc6865 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e2be9adc1a284f8b92e98861accc6865" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2be9adc1a284f8b92e98861accc6865" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 38543 } } }
I20260810 22:38:06.882560 5975 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2be9adc1a284f8b92e98861accc6865 [sys.catalog]: This master's current role is: LEADER
I20260810 22:38:06.882853 5976 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2be9adc1a284f8b92e98861accc6865 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2be9adc1a284f8b92e98861accc6865. Latest consensus state: current_term: 1 leader_uuid: "e2be9adc1a284f8b92e98861accc6865" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2be9adc1a284f8b92e98861accc6865" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 38543 } } }
I20260810 22:38:06.882966 5976 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2be9adc1a284f8b92e98861accc6865 [sys.catalog]: This master's current role is: LEADER
I20260810 22:38:06.883124 5984 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:38:06.886168 5984 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:38:06.891669 5984 catalog_manager.cc:1347] Generated new cluster ID: 9c8d715ef84842c3a8158fe689637b2a
I20260810 22:38:06.891758 5984 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:38:06.923425 5984 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:38:06.924860 5984 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:38:06.934155 5984 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e2be9adc1a284f8b92e98861accc6865: Generated new TSK 0
I20260810 22:38:06.935086 5984 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:38:06.940411 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786401215402979-17317-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786401215402979-17317-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786401215402979-17317-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.65:0
--local_ip_for_outbound_sockets=127.16.233.65
--webserver_interface=127.16.233.65
--webserver_port=0
--tserver_master_addrs=127.16.233.126:38543
--builtin_ntp_servers=127.16.233.84:36783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 22:38:07.062232 5994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:38:07.062598 5994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:38:07.062674 5994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:38:07.067003 5994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:38:07.067101 5994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:38:07.067195 5994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.65
I20260810 22:38:07.072294 5994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:36783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786401215402979-17317-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786401215402979-17317-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.233.65
--webserver_port=0
--tserver_master_addrs=127.16.233.126:38543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5994
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.65
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786401215402979-17317-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:38:07.073625 5994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:38:07.075198 5994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:38:07.086037 6002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:07.086021 6000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:07.086021 5999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:38:07.086292 5994 server_base.cc:1034] running on GCE node
I20260810 22:38:07.086740 5994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:38:07.087553 5994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:38:07.088786 5994 hybrid_clock.cc:648] HybridClock initialized: now 1786401487088759 us; error 51 us; skew 500 ppm
I20260810 22:38:07.090773 5994 webserver.cc:460] Webserver started at http://127.16.233.65:37867/ using document root <none> and password file <none>
I20260810 22:38:07.091396 5994 fs_manager.cc:359] Metadata directory not provided
I20260810 22:38:07.091497 5994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:38:07.091740 5994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:38:07.094008 5994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786401215402979-17317-0/minicluster-data/ts-0/data/instance:
uuid: "2babc051f64045f1a8fda2f9ec65e24e"
format_stamp: "Formatted at 2026-08-10 22:38:07 on dist-test-slave-77v9"
server_key: "73ec1ad596a99f57b7729383a1715682"
server_key_iv: "34273b566e1b8ef6d0597c60aba9bae2"
server_key_version: "encryptionkey@0"
I20260810 22:38:07.094605 5994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786401215402979-17317-0/minicluster-data/ts-0/wal/instance:
uuid: "2babc051f64045f1a8fda2f9ec65e24e"
format_stamp: "Formatted at 2026-08-10 22:38:07 on dist-test-slave-77v9"
server_key: "73ec1ad596a99f57b7729383a1715682"
server_key_iv: "34273b566e1b8ef6d0597c60aba9bae2"
server_key_version: "encryptionkey@0"
I20260810 22:38:07.098920 5994 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260810 22:38:07.101869 6009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:07.103209 5994 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 22:38:07.103381 5994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786401215402979-17317-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786401215402979-17317-0/minicluster-data/ts-0/wal
uuid: "2babc051f64045f1a8fda2f9ec65e24e"
format_stamp: "Formatted at 2026-08-10 22:38:07 on dist-test-slave-77v9"
server_key: "73ec1ad596a99f57b7729383a1715682"
server_key_iv: "34273b566e1b8ef6d0597c60aba9bae2"
server_key_version: "encryptionkey@0"
I20260810 22:38:07.103518 5994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786401215402979-17317-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786401215402979-17317-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786401215402979-17317-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:38:07.115847 5994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:38:07.116643 5994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:38:07.116899 5994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:38:07.117592 5994 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:38:07.118686 5994 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:38:07.118763 5994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:07.118840 5994 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:38:07.118884 5994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:07.128780 5994 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.65:33947
I20260810 22:38:07.128803 6130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.65:33947 every 8 connection(s)
I20260810 22:38:07.129954 5994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
I20260810 22:38:07.139441 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 5994
I20260810 22:38:07.139608 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786401215402979-17317-0/minicluster-data/ts-0/wal/instance
I20260810 22:38:07.140057 17317 external_mini_cluster.cc:1428] Setting key 59c630ffbc83b57d9d58b9a98b5b7ca8
I20260810 22:38:07.140443 5972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:07.132147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:36583 (local address 127.16.233.126:38543)
0810 22:38:07.132365 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:07.132368 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:07.133509 (+ 1141us) server_negotiation.cc:407] Connection header received
0810 22:38:07.135055 (+ 1546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:07.135059 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:07.135129 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:07.135247 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:07.136253 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:07.136907 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:07.138132 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:07.138392 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:07.138514 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:07.139859 (+ 1345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:07.139863 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:07.139867 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:38:07.139886 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:07.139914 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:38:07.139925 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:07.140012 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:07.140217 (+ 205us) server_negotiation.cc:299] Negotiation successful
0810 22:38:07.140294 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260810 22:38:07.141628 6133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:07.132555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:38543 (local address 127.16.233.65:36583)
0810 22:38:07.133327 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:07.133374 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 22:38:07.134803 (+ 1429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:38:07.135454 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:38:07.135461 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:38:07.135554 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:38:07.136093 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:07.136106 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:07.137061 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:07.137064 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:07.138000 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:07.138010 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:07.138569 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:07.138573 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:07.138754 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:07.139492 (+ 738us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 22:38:07.139493 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:38:07.139537 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 22:38:07.139642 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:38:07.140021 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:38:07.140030 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:38:07.140034 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 22:38:07.140138 (+ 104us) client_negotiation.cc:240] Negotiation successful
0810 22:38:07.140437 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":119,"threads_started":1}
I20260810 22:38:07.142400 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786401215402979-17317-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786401215402979-17317-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786401215402979-17317-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.66:0
--local_ip_for_outbound_sockets=127.16.233.66
--webserver_interface=127.16.233.66
--webserver_port=0
--tserver_master_addrs=127.16.233.126:38543
--builtin_ntp_servers=127.16.233.84:36783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 22:38:07.143050 6131 heartbeater.cc:347] Connected to a master server at 127.16.233.126:38543
I20260810 22:38:07.143312 6131 heartbeater.cc:464] Registering TS with master...
I20260810 22:38:07.143993 6131 heartbeater.cc:511] Master 127.16.233.126:38543 requested a full tablet report, sending...
I20260810 22:38:07.145897 5929 ts_manager.cc:194] Registered new tserver with Master: 2babc051f64045f1a8fda2f9ec65e24e (127.16.233.65:33947)
I20260810 22:38:07.147616 5929 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.233.65:36583
W20260810 22:38:07.264524 6134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:38:07.264894 6134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:38:07.265002 6134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:38:07.269316 6134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:38:07.269392 6134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:38:07.269451 6134 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.66
I20260810 22:38:07.274477 6134 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:36783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786401215402979-17317-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786401215402979-17317-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.233.66
--webserver_port=0
--tserver_master_addrs=127.16.233.126:38543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.66
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786401215402979-17317-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:38:07.275676 6134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:38:07.277223 6134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:38:07.287433 6143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:07.287441 6140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:07.287454 6141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:38:07.287887 6134 server_base.cc:1034] running on GCE node
I20260810 22:38:07.288342 6134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:38:07.289095 6134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:38:07.290295 6134 hybrid_clock.cc:648] HybridClock initialized: now 1786401487290278 us; error 35 us; skew 500 ppm
I20260810 22:38:07.292296 6134 webserver.cc:460] Webserver started at http://127.16.233.66:35849/ using document root <none> and password file <none>
I20260810 22:38:07.292956 6134 fs_manager.cc:359] Metadata directory not provided
I20260810 22:38:07.293030 6134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:38:07.293293 6134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:38:07.295455 6134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786401215402979-17317-0/minicluster-data/ts-1/data/instance:
uuid: "4e95ca28b5c046e3ad33bf2ea40b3c9e"
format_stamp: "Formatted at 2026-08-10 22:38:07 on dist-test-slave-77v9"
server_key: "abfd26f17eb8bd758a84b6e067e64ffd"
server_key_iv: "f09779f595ebc78aa90268c689b570a4"
server_key_version: "encryptionkey@0"
I20260810 22:38:07.296066 6134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786401215402979-17317-0/minicluster-data/ts-1/wal/instance:
uuid: "4e95ca28b5c046e3ad33bf2ea40b3c9e"
format_stamp: "Formatted at 2026-08-10 22:38:07 on dist-test-slave-77v9"
server_key: "abfd26f17eb8bd758a84b6e067e64ffd"
server_key_iv: "f09779f595ebc78aa90268c689b570a4"
server_key_version: "encryptionkey@0"
I20260810 22:38:07.300173 6134 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 22:38:07.303059 6152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:07.304595 6134 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 22:38:07.304878 6134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786401215402979-17317-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786401215402979-17317-0/minicluster-data/ts-1/wal
uuid: "4e95ca28b5c046e3ad33bf2ea40b3c9e"
format_stamp: "Formatted at 2026-08-10 22:38:07 on dist-test-slave-77v9"
server_key: "abfd26f17eb8bd758a84b6e067e64ffd"
server_key_iv: "f09779f595ebc78aa90268c689b570a4"
server_key_version: "encryptionkey@0"
I20260810 22:38:07.305022 6134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786401215402979-17317-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786401215402979-17317-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786401215402979-17317-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:38:07.330894 6134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:38:07.331737 6134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:38:07.331974 6134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:38:07.332643 6134 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:38:07.333742 6134 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:38:07.333796 6134 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:07.333861 6134 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:38:07.333895 6134 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:07.343976 6134 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.66:36463
I20260810 22:38:07.344012 6276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.66:36463 every 8 connection(s)
I20260810 22:38:07.345057 6134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
I20260810 22:38:07.352177 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 6134
I20260810 22:38:07.352300 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786401215402979-17317-0/minicluster-data/ts-1/wal/instance
I20260810 22:38:07.352610 17317 external_mini_cluster.cc:1428] Setting key 81d70cdb5492975fa0ae9cca4dcc65d7
I20260810 22:38:07.355101 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786401215402979-17317-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786401215402979-17317-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786401215402979-17317-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.67:0
--local_ip_for_outbound_sockets=127.16.233.67
--webserver_interface=127.16.233.67
--webserver_port=0
--tserver_master_addrs=127.16.233.126:38543
--builtin_ntp_servers=127.16.233.84:36783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 22:38:07.356031 5972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:07.347159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:59853 (local address 127.16.233.126:38543)
0810 22:38:07.347359 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:07.347364 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:07.348566 (+ 1202us) server_negotiation.cc:407] Connection header received
0810 22:38:07.350130 (+ 1564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:07.350133 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:07.350194 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:07.350300 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:07.351224 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:07.351891 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:07.353546 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:07.353783 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:07.353905 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:07.355498 (+ 1593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:07.355501 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:07.355503 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:38:07.355515 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:07.355536 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:38:07.355544 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:07.355632 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:07.355809 (+ 177us) server_negotiation.cc:299] Negotiation successful
0810 22:38:07.355882 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 22:38:07.357129 6279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:07.347675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:38543 (local address 127.16.233.66:59853)
0810 22:38:07.348404 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:07.348449 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 22:38:07.349888 (+ 1439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:38:07.350481 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:38:07.350487 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:38:07.350570 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:38:07.351068 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:07.351080 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:07.352078 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:07.352081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:07.353406 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:07.353417 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:07.353953 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:07.353958 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:07.354205 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:07.355101 (+ 896us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 22:38:07.355103 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:38:07.355148 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 22:38:07.355282 (+ 134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:38:07.355635 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:38:07.355648 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:38:07.355657 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 22:38:07.355809 (+ 152us) client_negotiation.cc:240] Negotiation successful
0810 22:38:07.356161 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":130,"threads_started":1}
I20260810 22:38:07.358500 6277 heartbeater.cc:347] Connected to a master server at 127.16.233.126:38543
I20260810 22:38:07.358834 6277 heartbeater.cc:464] Registering TS with master...
I20260810 22:38:07.359407 6277 heartbeater.cc:511] Master 127.16.233.126:38543 requested a full tablet report, sending...
I20260810 22:38:07.360620 5929 ts_manager.cc:194] Registered new tserver with Master: 4e95ca28b5c046e3ad33bf2ea40b3c9e (127.16.233.66:36463)
I20260810 22:38:07.361552 5929 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.233.66:59853
W20260810 22:38:07.475967 6281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:38:07.476285 6281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:38:07.476328 6281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:38:07.480500 6281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:38:07.480585 6281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:38:07.480645 6281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.67
I20260810 22:38:07.486889 6281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:36783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786401215402979-17317-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786401215402979-17317-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.233.67
--webserver_port=0
--tserver_master_addrs=127.16.233.126:38543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.67
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786401215402979-17317-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:38:07.488119 6281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:38:07.489712 6281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:38:07.501210 6289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:07.501199 6286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:07.501204 6287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:38:07.501387 6281 server_base.cc:1034] running on GCE node
I20260810 22:38:07.502204 6281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:38:07.503103 6281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:38:07.504348 6281 hybrid_clock.cc:648] HybridClock initialized: now 1786401487504305 us; error 49 us; skew 500 ppm
I20260810 22:38:07.506837 6281 webserver.cc:460] Webserver started at http://127.16.233.67:36841/ using document root <none> and password file <none>
I20260810 22:38:07.507519 6281 fs_manager.cc:359] Metadata directory not provided
I20260810 22:38:07.507635 6281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:38:07.507933 6281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:38:07.510418 6281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786401215402979-17317-0/minicluster-data/ts-2/data/instance:
uuid: "5de3b8f51b5f4c309c79028c23043efb"
format_stamp: "Formatted at 2026-08-10 22:38:07 on dist-test-slave-77v9"
server_key: "7dac95e972b24aec81b26387d4fb8cc2"
server_key_iv: "fd446e254c92400b8fe7ca966470bcf8"
server_key_version: "encryptionkey@0"
I20260810 22:38:07.511135 6281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786401215402979-17317-0/minicluster-data/ts-2/wal/instance:
uuid: "5de3b8f51b5f4c309c79028c23043efb"
format_stamp: "Formatted at 2026-08-10 22:38:07 on dist-test-slave-77v9"
server_key: "7dac95e972b24aec81b26387d4fb8cc2"
server_key_iv: "fd446e254c92400b8fe7ca966470bcf8"
server_key_version: "encryptionkey@0"
I20260810 22:38:07.515843 6281 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260810 22:38:07.519042 6295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:07.520491 6281 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 22:38:07.520669 6281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786401215402979-17317-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786401215402979-17317-0/minicluster-data/ts-2/wal
uuid: "5de3b8f51b5f4c309c79028c23043efb"
format_stamp: "Formatted at 2026-08-10 22:38:07 on dist-test-slave-77v9"
server_key: "7dac95e972b24aec81b26387d4fb8cc2"
server_key_iv: "fd446e254c92400b8fe7ca966470bcf8"
server_key_version: "encryptionkey@0"
I20260810 22:38:07.520965 6281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786401215402979-17317-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786401215402979-17317-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786401215402979-17317-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:38:07.540961 6281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:38:07.541833 6281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:38:07.542073 6281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:38:07.542771 6281 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:38:07.543960 6281 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:38:07.544044 6281 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:07.544124 6281 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:38:07.544171 6281 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:07.555171 6281 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.67:45363
I20260810 22:38:07.555202 6423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.67:45363 every 8 connection(s)
I20260810 22:38:07.556233 6281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
I20260810 22:38:07.563727 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 6281
I20260810 22:38:07.563902 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786401215402979-17317-0/minicluster-data/ts-2/wal/instance
I20260810 22:38:07.564249 17317 external_mini_cluster.cc:1428] Setting key 5786bfc3589860c6ab9849adfed1a6e8
I20260810 22:38:07.567135 5972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:07.558403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:55315 (local address 127.16.233.126:38543)
0810 22:38:07.558578 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:07.558582 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:07.560007 (+ 1425us) server_negotiation.cc:407] Connection header received
0810 22:38:07.561546 (+ 1539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:07.561549 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:07.561615 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:07.561744 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:07.562745 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:07.563289 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:07.564640 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:07.564982 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:07.565104 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:07.566613 (+ 1509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:07.566617 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:07.566620 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:38:07.566633 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:07.566660 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:38:07.566668 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:07.566754 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:07.566960 (+ 206us) server_negotiation.cc:299] Negotiation successful
0810 22:38:07.567017 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 22:38:07.568127 6426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:07.559052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:38543 (local address 127.16.233.67:55315)
0810 22:38:07.559787 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:07.559831 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 22:38:07.561306 (+ 1475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:38:07.561944 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:38:07.561951 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:38:07.562042 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:38:07.562562 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:07.562575 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:07.563429 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:07.563434 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:07.564503 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:07.564514 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:07.565301 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:07.565304 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:07.565524 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:07.566277 (+ 753us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 22:38:07.566279 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:38:07.566325 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 22:38:07.566396 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:38:07.566795 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:38:07.566804 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:38:07.566812 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 22:38:07.566944 (+ 132us) client_negotiation.cc:240] Negotiation successful
0810 22:38:07.567237 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":158,"threads_started":1}
I20260810 22:38:07.569425 6424 heartbeater.cc:347] Connected to a master server at 127.16.233.126:38543
I20260810 22:38:07.569680 6424 heartbeater.cc:464] Registering TS with master...
I20260810 22:38:07.570282 6424 heartbeater.cc:511] Master 127.16.233.126:38543 requested a full tablet report, sending...
I20260810 22:38:07.571381 5929 ts_manager.cc:194] Registered new tserver with Master: 5de3b8f51b5f4c309c79028c23043efb (127.16.233.67:45363)
I20260810 22:38:07.572022 5929 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.233.67:55315
I20260810 22:38:07.579103 17317 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 22:38:07.586133 5972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:07.581392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41414 (local address 127.16.233.126:38543)
0810 22:38:07.581554 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:07.581560 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:07.581726 (+ 166us) server_negotiation.cc:407] Connection header received
0810 22:38:07.581797 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:07.581799 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:07.581909 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:07.582059 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:07.582463 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:07.582992 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:07.583956 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:07.584176 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:07.584261 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:07.585381 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:07.585384 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:07.585386 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:38:07.585401 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:07.585423 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:38:07.585431 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:07.585557 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:07.585880 (+ 323us) server_negotiation.cc:299] Negotiation successful
0810 22:38:07.585944 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 22:38:07.590178 5929 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41414:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 22:38:07.593710 5929 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:38:07.611613 6437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:07.605002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.65:33947 (local address 127.0.0.1:38442)
0810 22:38:07.605478 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:07.605495 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 22:38:07.605596 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:38:07.606247 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:38:07.606253 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:38:07.606278 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:38:07.606471 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:07.606480 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:07.607752 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:07.607757 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:07.609751 (+ 1994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:07.609790 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:07.610893 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:07.610898 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:07.611055 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:07.611128 (+ 73us) client_negotiation.cc:769] Sending connection context
0810 22:38:07.611255 (+ 127us) client_negotiation.cc:240] Negotiation successful
0810 22:38:07.611396 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"mutex_wait_us":208,"thread_start_us":96,"threads_started":1}
I20260810 22:38:07.611791 6438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:07.604299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53550 (local address 127.16.233.66:36463)
0810 22:38:07.605921 (+ 1622us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:07.605937 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:07.605966 (+ 29us) server_negotiation.cc:407] Connection header received
0810 22:38:07.606061 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:07.606067 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:07.606279 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:07.606471 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:38:07.606736 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:07.608184 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:07.609719 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:07.610594 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:07.610711 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:07.610929 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:07.611407 (+ 478us) server_negotiation.cc:299] Negotiation successful
0810 22:38:07.611517 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1497,"thread_start_us":145,"threads_started":1}
I20260810 22:38:07.612355 6439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:07.604626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54266 (local address 127.16.233.67:45363)
0810 22:38:07.605623 (+ 997us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:07.605990 (+ 367us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:07.606018 (+ 28us) server_negotiation.cc:407] Connection header received
0810 22:38:07.606097 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:07.606103 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:07.606278 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:07.606471 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:38:07.607023 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:07.608296 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:07.610285 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:07.611033 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:07.611162 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:07.611379 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:07.612024 (+ 645us) server_negotiation.cc:299] Negotiation successful
0810 22:38:07.612149 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":878,"thread_start_us":192,"threads_started":1}
I20260810 22:38:07.612644 6441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:07.605267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38442 (local address 127.16.233.65:33947)
0810 22:38:07.605712 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:07.605719 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:07.605747 (+ 28us) server_negotiation.cc:407] Connection header received
0810 22:38:07.605843 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:07.605848 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:07.606073 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:07.606255 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:38:07.606622 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:07.607622 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:07.609959 (+ 2337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:07.610750 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:07.610854 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:07.611010 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:07.612193 (+ 1183us) server_negotiation.cc:299] Negotiation successful
0810 22:38:07.612313 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":132,"threads_started":1}
I20260810 22:38:07.611914 6440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:07.604173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.67:45363 (local address 127.0.0.1:54266)
0810 22:38:07.605334 (+ 1161us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:07.605351 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 22:38:07.605703 (+ 352us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:38:07.606663 (+ 960us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:38:07.606668 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:38:07.606692 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:38:07.606815 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:07.606823 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:07.608438 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:07.608441 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:07.610138 (+ 1697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:07.610147 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:07.611453 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:07.611458 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:07.611558 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:07.611583 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 22:38:07.611671 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 22:38:07.611750 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1072,"mutex_wait_us":3,"thread_start_us":128,"threads_started":1}
I20260810 22:38:07.612088 6436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:07.604144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.66:36463 (local address 127.0.0.1:53550)
0810 22:38:07.604909 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:07.604970 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 22:38:07.605141 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:38:07.606441 (+ 1300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:38:07.606444 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:38:07.606486 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:38:07.606629 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:07.606637 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:07.608368 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:07.608376 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:07.609563 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:07.609573 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:07.610998 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:07.611002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:07.611207 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:07.611244 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 22:38:07.611345 (+ 101us) client_negotiation.cc:240] Negotiation successful
0810 22:38:07.611424 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"mutex_wait_us":178,"thread_start_us":113,"threads_started":1}
I20260810 22:38:07.616042 6350 tablet_service.cc:1467] Processing CreateTablet for tablet 8c7ecedde73b472084bd9925f2b76d9f (DEFAULT_TABLE table=test-table [id=04b847cfa420493c8f9fe0dd484b5669]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:38:07.616178 6063 tablet_service.cc:1467] Processing CreateTablet for tablet 8c7ecedde73b472084bd9925f2b76d9f (DEFAULT_TABLE table=test-table [id=04b847cfa420493c8f9fe0dd484b5669]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:38:07.616178 6205 tablet_service.cc:1467] Processing CreateTablet for tablet 8c7ecedde73b472084bd9925f2b76d9f (DEFAULT_TABLE table=test-table [id=04b847cfa420493c8f9fe0dd484b5669]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:38:07.617343 6350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c7ecedde73b472084bd9925f2b76d9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:38:07.617396 6205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c7ecedde73b472084bd9925f2b76d9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:38:07.617396 6063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c7ecedde73b472084bd9925f2b76d9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:38:07.624097 6442 tablet_bootstrap.cc:492] T 8c7ecedde73b472084bd9925f2b76d9f P 5de3b8f51b5f4c309c79028c23043efb: Bootstrap starting.
I20260810 22:38:07.626631 6443 tablet_bootstrap.cc:492] T 8c7ecedde73b472084bd9925f2b76d9f P 2babc051f64045f1a8fda2f9ec65e24e: Bootstrap starting.
I20260810 22:38:07.627048 6442 tablet_bootstrap.cc:654] T 8c7ecedde73b472084bd9925f2b76d9f P 5de3b8f51b5f4c309c79028c23043efb: Neither blocks nor log segments found. Creating new log.
I20260810 22:38:07.628121 6442 log.cc:824] T 8c7ecedde73b472084bd9925f2b76d9f P 5de3b8f51b5f4c309c79028c23043efb: Log is configured to *not* fsync() on all Append() calls
I20260810 22:38:07.629235 6443 tablet_bootstrap.cc:654] T 8c7ecedde73b472084bd9925f2b76d9f P 2babc051f64045f1a8fda2f9ec65e24e: Neither blocks nor log segments found. Creating new log.
I20260810 22:38:07.629318 6444 tablet_bootstrap.cc:492] T 8c7ecedde73b472084bd9925f2b76d9f P 4e95ca28b5c046e3ad33bf2ea40b3c9e: Bootstrap starting.
I20260810 22:38:07.630139 6442 tablet_bootstrap.cc:492] T 8c7ecedde73b472084bd9925f2b76d9f P 5de3b8f51b5f4c309c79028c23043efb: No bootstrap required, opened a new log
I20260810 22:38:07.630198 6443 log.cc:824] T 8c7ecedde73b472084bd9925f2b76d9f P 2babc051f64045f1a8fda2f9ec65e24e: Log is configured to *not* fsync() on all Append() calls
I20260810 22:38:07.630388 6442 ts_tablet_manager.cc:1397] T 8c7ecedde73b472084bd9925f2b76d9f P 5de3b8f51b5f4c309c79028c23043efb: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260810 22:38:07.631870 6444 tablet_bootstrap.cc:654] T 8c7ecedde73b472084bd9925f2b76d9f P 4e95ca28b5c046e3ad33bf2ea40b3c9e: Neither blocks nor log segments found. Creating new log.
I20260810 22:38:07.632843 6443 tablet_bootstrap.cc:492] T 8c7ecedde73b472084bd9925f2b76d9f P 2babc051f64045f1a8fda2f9ec65e24e: No bootstrap required, opened a new log
I20260810 22:38:07.632853 6444 log.cc:824] T 8c7ecedde73b472084bd9925f2b76d9f P 4e95ca28b5c046e3ad33bf2ea40b3c9e: Log is configured to *not* fsync() on all Append() calls
I20260810 22:38:07.633077 6443 ts_tablet_manager.cc:1397] T 8c7ecedde73b472084bd9925f2b76d9f P 2babc051f64045f1a8fda2f9ec65e24e: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260810 22:38:07.634374 6442 raft_consensus.cc:348] T 8c7ecedde73b472084bd9925f2b76d9f P 5de3b8f51b5f4c309c79028c23043efb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e95ca28b5c046e3ad33bf2ea40b3c9e" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 36463 } } peers { permanent_uuid: "5de3b8f51b5f4c309c79028c23043efb" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 45363 } } peers { permanent_uuid: "2babc051f64045f1a8fda2f9ec65e24e" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33947 } }
I20260810 22:38:07.634658 6442 raft_consensus.cc:374] T 8c7ecedde73b472084bd9925f2b76d9f P 5de3b8f51b5f4c309c79028c23043efb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:38:07.634711 6442 raft_consensus.cc:729] T 8c7ecedde73b472084bd9925f2b76d9f P 5de3b8f51b5f4c309c79028c23043efb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5de3b8f51b5f4c309c79028c23043efb, State: Initialized, Role: FOLLOWER
I20260810 22:38:07.635234 6442 consensus_queue.cc:260] T 8c7ecedde73b472084bd9925f2b76d9f P 5de3b8f51b5f4c309c79028c23043efb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e95ca28b5c046e3ad33bf2ea40b3c9e" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 36463 } } peers { permanent_uuid: "5de3b8f51b5f4c309c79028c23043efb" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 45363 } } peers { permanent_uuid: "2babc051f64045f1a8fda2f9ec65e24e" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33947 } }
I20260810 22:38:07.635949 6424 heartbeater.cc:503] Master 127.16.233.126:38543 was elected leader, sending a full tablet report...
I20260810 22:38:07.635986 6444 tablet_bootstrap.cc:492] T 8c7ecedde73b472084bd9925f2b76d9f P 4e95ca28b5c046e3ad33bf2ea40b3c9e: No bootstrap required, opened a new log
I20260810 22:38:07.636212 6444 ts_tablet_manager.cc:1397] T 8c7ecedde73b472084bd9925f2b76d9f P 4e95ca28b5c046e3ad33bf2ea40b3c9e: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260810 22:38:07.636313 6442 ts_tablet_manager.cc:1428] T 8c7ecedde73b472084bd9925f2b76d9f P 5de3b8f51b5f4c309c79028c23043efb: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 22:38:07.638227 6443 raft_consensus.cc:348] T 8c7ecedde73b472084bd9925f2b76d9f P 2babc051f64045f1a8fda2f9ec65e24e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e95ca28b5c046e3ad33bf2ea40b3c9e" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 36463 } } peers { permanent_uuid: "5de3b8f51b5f4c309c79028c23043efb" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 45363 } } peers { permanent_uuid: "2babc051f64045f1a8fda2f9ec65e24e" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33947 } }
I20260810 22:38:07.638552 6443 raft_consensus.cc:374] T 8c7ecedde73b472084bd9925f2b76d9f P 2babc051f64045f1a8fda2f9ec65e24e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:38:07.638621 6443 raft_consensus.cc:729] T 8c7ecedde73b472084bd9925f2b76d9f P 2babc051f64045f1a8fda2f9ec65e24e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2babc051f64045f1a8fda2f9ec65e24e, State: Initialized, Role: FOLLOWER
I20260810 22:38:07.639156 6443 consensus_queue.cc:260] T 8c7ecedde73b472084bd9925f2b76d9f P 2babc051f64045f1a8fda2f9ec65e24e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e95ca28b5c046e3ad33bf2ea40b3c9e" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 36463 } } peers { permanent_uuid: "5de3b8f51b5f4c309c79028c23043efb" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 45363 } } peers { permanent_uuid: "2babc051f64045f1a8fda2f9ec65e24e" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33947 } }
I20260810 22:38:07.640136 6131 heartbeater.cc:503] Master 127.16.233.126:38543 was elected leader, sending a full tablet report...
I20260810 22:38:07.640369 6443 ts_tablet_manager.cc:1428] T 8c7ecedde73b472084bd9925f2b76d9f P 2babc051f64045f1a8fda2f9ec65e24e: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260810 22:38:07.641410 6444 raft_consensus.cc:348] T 8c7ecedde73b472084bd9925f2b76d9f P 4e95ca28b5c046e3ad33bf2ea40b3c9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e95ca28b5c046e3ad33bf2ea40b3c9e" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 36463 } } peers { permanent_uuid: "5de3b8f51b5f4c309c79028c23043efb" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 45363 } } peers { permanent_uuid: "2babc051f64045f1a8fda2f9ec65e24e" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33947 } }
I20260810 22:38:07.641695 6444 raft_consensus.cc:374] T 8c7ecedde73b472084bd9925f2b76d9f P 4e95ca28b5c046e3ad33bf2ea40b3c9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:38:07.641804 6444 raft_consensus.cc:729] T 8c7ecedde73b472084bd9925f2b76d9f P 4e95ca28b5c046e3ad33bf2ea40b3c9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e95ca28b5c046e3ad33bf2ea40b3c9e, State: Initialized, Role: FOLLOWER
I20260810 22:38:07.642347 6444 consensus_queue.cc:260] T 8c7ecedde73b472084bd9925f2b76d9f P 4e95ca28b5c046e3ad33bf2ea40b3c9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e95ca28b5c046e3ad33bf2ea40b3c9e" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 36463 } } peers { permanent_uuid: "5de3b8f51b5f4c309c79028c23043efb" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 45363 } } peers { permanent_uuid: "2babc051f64045f1a8fda2f9ec65e24e" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33947 } }
I20260810 22:38:07.643141 6277 heartbeater.cc:503] Master 127.16.233.126:38543 was elected leader, sending a full tablet report...
I20260810 22:38:07.643574 6444 ts_tablet_manager.cc:1428] T 8c7ecedde73b472084bd9925f2b76d9f P 4e95ca28b5c046e3ad33bf2ea40b3c9e: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
W20260810 22:38:07.807452 6425 tablet.cc:2369] T 8c7ecedde73b472084bd9925f2b76d9f P 5de3b8f51b5f4c309c79028c23043efb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 22:38:07.846369 6278 tablet.cc:2369] T 8c7ecedde73b472084bd9925f2b76d9f P 4e95ca28b5c046e3ad33bf2ea40b3c9e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:38:07.873168 6448 raft_consensus.cc:482] T 8c7ecedde73b472084bd9925f2b76d9f P 5de3b8f51b5f4c309c79028c23043efb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:38:07.873415 6448 raft_consensus.cc:504] T 8c7ecedde73b472084bd9925f2b76d9f P 5de3b8f51b5f4c309c79028c23043efb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e95ca28b5c046e3ad33bf2ea40b3c9e" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 36463 } } peers { permanent_uuid: "5de3b8f51b5f4c309c79028c23043efb" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 45363 } } peers { permanent_uuid: "2babc051f64045f1a8fda2f9ec65e24e" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33947 } }
I20260810 22:38:07.874750 6448 leader_election.cc:288] T 8c7ecedde73b472084bd9925f2b76d9f P 5de3b8f51b5f4c309c79028c23043efb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e95ca28b5c046e3ad33bf2ea40b3c9e (127.16.233.66:36463), 2babc051f64045f1a8fda2f9ec65e24e (127.16.233.65:33947)
I20260810 22:38:07.880780 6441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:07.875407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:47603 (local address 127.16.233.65:33947)
0810 22:38:07.875557 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:07.875563 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:07.875795 (+ 232us) server_negotiation.cc:407] Connection header received
0810 22:38:07.875875 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:07.875878 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:07.875933 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:07.876003 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:38:07.876312 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:07.877145 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:07.879255 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:07.879930 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:07.880092 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:07.880148 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:07.880551 (+ 403us) server_negotiation.cc:299] Negotiation successful
0810 22:38:07.880639 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 22:38:07.880860 6426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:07.875061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.66:36463 (local address 127.16.233.67:53571)
0810 22:38:07.875229 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:07.875250 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 22:38:07.875362 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:38:07.875731 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:38:07.875735 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:38:07.875761 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:38:07.875933 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:07.875941 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:07.877186 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:07.877189 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:07.879008 (+ 1819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:07.879015 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:07.880113 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:07.880117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:07.880311 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:07.880357 (+ 46us) client_negotiation.cc:769] Sending connection context
0810 22:38:07.880451 (+ 94us) client_negotiation.cc:240] Negotiation successful
0810 22:38:07.880588 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":266}
I20260810 22:38:07.880893 6438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:07.875289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:53571 (local address 127.16.233.66:36463)
0810 22:38:07.875440 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:07.875445 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:07.875464 (+ 19us) server_negotiation.cc:407] Connection header received
0810 22:38:07.875532 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:07.875536 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:07.875598 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:07.875674 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:38:07.876097 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:07.877036 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:07.879212 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:07.879954 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:07.880093 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:07.880148 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:07.880552 (+ 404us) server_negotiation.cc:299] Negotiation successful
0810 22:38:07.880639 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 22:38:07.881132 6451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:07.875216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.65:33947 (local address 127.16.233.67:47603)
0810 22:38:07.875639 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:07.875681 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 22:38:07.875780 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:38:07.876047 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:38:07.876053 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:38:07.876076 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:38:07.876185 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:07.876189 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:07.877289 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:07.877293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:07.878986 (+ 1693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:07.879000 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:07.880113 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:07.880117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:07.880308 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:07.880371 (+ 63us) client_negotiation.cc:769] Sending connection context
0810 22:38:07.880451 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 22:38:07.880584 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"mutex_wait_us":158,"thread_start_us":140,"threads_started":1}
W20260810 22:38:07.881526 6132 tablet.cc:2369] T 8c7ecedde73b472084bd9925f2b76d9f P 2babc051f64045f1a8fda2f9ec65e24e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:38:07.881770 6226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c7ecedde73b472084bd9925f2b76d9f" candidate_uuid: "5de3b8f51b5f4c309c79028c23043efb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e95ca28b5c046e3ad33bf2ea40b3c9e" is_pre_election: true
I20260810 22:38:07.881953 6083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c7ecedde73b472084bd9925f2b76d9f" candidate_uuid: "5de3b8f51b5f4c309c79028c23043efb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2babc051f64045f1a8fda2f9ec65e24e" is_pre_election: true
I20260810 22:38:07.882123 6226 raft_consensus.cc:2454] T 8c7ecedde73b472084bd9925f2b76d9f P 4e95ca28b5c046e3ad33bf2ea40b3c9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5de3b8f51b5f4c309c79028c23043efb in term 0.
I20260810 22:38:07.882332 6083 raft_consensus.cc:2454] T 8c7ecedde73b472084bd9925f2b76d9f P 2babc051f64045f1a8fda2f9ec65e24e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5de3b8f51b5f4c309c79028c23043efb in term 0.
I20260810 22:38:07.882728 6299 leader_election.cc:302] T 8c7ecedde73b472084bd9925f2b76d9f P 5de3b8f51b5f4c309c79028c23043efb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e95ca28b5c046e3ad33bf2ea40b3c9e, 5de3b8f51b5f4c309c79028c23043efb; no voters:
I20260810 22:38:07.883126 6448 raft_consensus.cc:2781] T 8c7ecedde73b472084bd9925f2b76d9f P 5de3b8f51b5f4c309c79028c23043efb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:38:07.883222 6448 raft_consensus.cc:482] T 8c7ecedde73b472084bd9925f2b76d9f P 5de3b8f51b5f4c309c79028c23043efb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:38:07.883257 6448 raft_consensus.cc:3037] T 8c7ecedde73b472084bd9925f2b76d9f P 5de3b8f51b5f4c309c79028c23043efb [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:38:07.884549 6448 raft_consensus.cc:504] T 8c7ecedde73b472084bd9925f2b76d9f P 5de3b8f51b5f4c309c79028c23043efb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e95ca28b5c046e3ad33bf2ea40b3c9e" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 36463 } } peers { permanent_uuid: "5de3b8f51b5f4c309c79028c23043efb" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 45363 } } peers { permanent_uuid: "2babc051f64045f1a8fda2f9ec65e24e" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33947 } }
I20260810 22:38:07.885030 6448 leader_election.cc:288] T 8c7ecedde73b472084bd9925f2b76d9f P 5de3b8f51b5f4c309c79028c23043efb [CANDIDATE]: Term 1 election: Requested vote from peers 4e95ca28b5c046e3ad33bf2ea40b3c9e (127.16.233.66:36463), 2babc051f64045f1a8fda2f9ec65e24e (127.16.233.65:33947)
I20260810 22:38:07.885481 6083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c7ecedde73b472084bd9925f2b76d9f" candidate_uuid: "5de3b8f51b5f4c309c79028c23043efb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2babc051f64045f1a8fda2f9ec65e24e"
I20260810 22:38:07.885495 6226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c7ecedde73b472084bd9925f2b76d9f" candidate_uuid: "5de3b8f51b5f4c309c79028c23043efb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e95ca28b5c046e3ad33bf2ea40b3c9e"
I20260810 22:38:07.885600 6083 raft_consensus.cc:3037] T 8c7ecedde73b472084bd9925f2b76d9f P 2babc051f64045f1a8fda2f9ec65e24e [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:38:07.885637 6226 raft_consensus.cc:3037] T 8c7ecedde73b472084bd9925f2b76d9f P 4e95ca28b5c046e3ad33bf2ea40b3c9e [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:38:07.886737 6083 raft_consensus.cc:2454] T 8c7ecedde73b472084bd9925f2b76d9f P 2babc051f64045f1a8fda2f9ec65e24e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5de3b8f51b5f4c309c79028c23043efb in term 1.
I20260810 22:38:07.886744 6226 raft_consensus.cc:2454] T 8c7ecedde73b472084bd9925f2b76d9f P 4e95ca28b5c046e3ad33bf2ea40b3c9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5de3b8f51b5f4c309c79028c23043efb in term 1.
I20260810 22:38:07.887121 6297 leader_election.cc:302] T 8c7ecedde73b472084bd9925f2b76d9f P 5de3b8f51b5f4c309c79028c23043efb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2babc051f64045f1a8fda2f9ec65e24e, 5de3b8f51b5f4c309c79028c23043efb; no voters:
I20260810 22:38:07.887312 6448 raft_consensus.cc:2781] T 8c7ecedde73b472084bd9925f2b76d9f P 5de3b8f51b5f4c309c79028c23043efb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:38:07.887579 6448 raft_consensus.cc:686] T 8c7ecedde73b472084bd9925f2b76d9f P 5de3b8f51b5f4c309c79028c23043efb [term 1 LEADER]: Becoming Leader. State: Replica: 5de3b8f51b5f4c309c79028c23043efb, State: Running, Role: LEADER
I20260810 22:38:07.887928 6448 consensus_queue.cc:237] T 8c7ecedde73b472084bd9925f2b76d9f P 5de3b8f51b5f4c309c79028c23043efb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e95ca28b5c046e3ad33bf2ea40b3c9e" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 36463 } } peers { permanent_uuid: "5de3b8f51b5f4c309c79028c23043efb" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 45363 } } peers { permanent_uuid: "2babc051f64045f1a8fda2f9ec65e24e" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33947 } }
I20260810 22:38:07.891541 5928 catalog_manager.cc:5495] T 8c7ecedde73b472084bd9925f2b76d9f P 5de3b8f51b5f4c309c79028c23043efb reported cstate change: term changed from 0 to 1, leader changed from <none> to 5de3b8f51b5f4c309c79028c23043efb (127.16.233.67). New cstate: current_term: 1 leader_uuid: "5de3b8f51b5f4c309c79028c23043efb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e95ca28b5c046e3ad33bf2ea40b3c9e" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 36463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5de3b8f51b5f4c309c79028c23043efb" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 45363 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2babc051f64045f1a8fda2f9ec65e24e" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33947 } health_report { overall_health: UNKNOWN } } }
I20260810 22:38:07.909658 6439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:07.905489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54272 (local address 127.16.233.67:45363)
0810 22:38:07.905646 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:07.905651 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:07.905668 (+ 17us) server_negotiation.cc:407] Connection header received
0810 22:38:07.905808 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:07.905810 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:07.905871 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:07.906003 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:38:07.906268 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:07.906842 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:07.907855 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:07.908145 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:07.908262 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:07.908623 (+ 361us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:38:07.909135 (+ 512us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:38:07.909310 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:07.909425 (+ 115us) server_negotiation.cc:299] Negotiation successful
0810 22:38:07.909468 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 22:38:07.916558 6226 raft_consensus.cc:1261] T 8c7ecedde73b472084bd9925f2b76d9f P 4e95ca28b5c046e3ad33bf2ea40b3c9e [term 1 FOLLOWER]: Refusing update from remote peer 5de3b8f51b5f4c309c79028c23043efb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:38:07.916558 6083 raft_consensus.cc:1261] T 8c7ecedde73b472084bd9925f2b76d9f P 2babc051f64045f1a8fda2f9ec65e24e [term 1 FOLLOWER]: Refusing update from remote peer 5de3b8f51b5f4c309c79028c23043efb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:38:07.917546 6448 consensus_queue.cc:1035] T 8c7ecedde73b472084bd9925f2b76d9f P 5de3b8f51b5f4c309c79028c23043efb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e95ca28b5c046e3ad33bf2ea40b3c9e" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 36463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:38:07.917801 6452 consensus_queue.cc:1035] T 8c7ecedde73b472084bd9925f2b76d9f P 5de3b8f51b5f4c309c79028c23043efb [LEADER]: Connected to new peer: Peer: permanent_uuid: "2babc051f64045f1a8fda2f9ec65e24e" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 33947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:38:07.929764 6457 mvcc.cc:205] Tried to move back new op lower bound from 7317100494497816576 to 7317100494391558144. Current Snapshot: MvccSnapshot[applied={T|T < 7317100494497816576}]
I20260810 22:38:07.940526 5929 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41414:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:38:07.940747 5929 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41414:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:38:07.943617 5929 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e2be9adc1a284f8b92e98861accc6865: Sending DeleteTablet for 3 replicas of tablet 8c7ecedde73b472084bd9925f2b76d9f
I20260810 22:38:07.944667 6063 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c7ecedde73b472084bd9925f2b76d9f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:38:07 UTC) from {username='slave'} at 127.0.0.1:38442
I20260810 22:38:07.944718 6350 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c7ecedde73b472084bd9925f2b76d9f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:38:07 UTC) from {username='slave'} at 127.0.0.1:54266
I20260810 22:38:07.944667 6205 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c7ecedde73b472084bd9925f2b76d9f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:38:07 UTC) from {username='slave'} at 127.0.0.1:53550
I20260810 22:38:07.945362 6468 tablet_replica.cc:330] T 8c7ecedde73b472084bd9925f2b76d9f P 5de3b8f51b5f4c309c79028c23043efb: stopping tablet replica
I20260810 22:38:07.945391 6469 tablet_replica.cc:330] T 8c7ecedde73b472084bd9925f2b76d9f P 4e95ca28b5c046e3ad33bf2ea40b3c9e: stopping tablet replica
I20260810 22:38:07.945752 6469 raft_consensus.cc:2229] T 8c7ecedde73b472084bd9925f2b76d9f P 4e95ca28b5c046e3ad33bf2ea40b3c9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:38:07.945753 6468 raft_consensus.cc:2229] T 8c7ecedde73b472084bd9925f2b76d9f P 5de3b8f51b5f4c309c79028c23043efb [term 1 LEADER]: Raft consensus shutting down.
I20260810 22:38:07.946218 6468 raft_consensus.cc:2258] T 8c7ecedde73b472084bd9925f2b76d9f P 5de3b8f51b5f4c309c79028c23043efb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:38:07.946231 6467 tablet_replica.cc:330] T 8c7ecedde73b472084bd9925f2b76d9f P 2babc051f64045f1a8fda2f9ec65e24e: stopping tablet replica
I20260810 22:38:07.946346 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 5994
I20260810 22:38:07.946632 6467 raft_consensus.cc:2229] T 8c7ecedde73b472084bd9925f2b76d9f P 2babc051f64045f1a8fda2f9ec65e24e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:38:07.953418 6469 raft_consensus.cc:2258] T 8c7ecedde73b472084bd9925f2b76d9f P 4e95ca28b5c046e3ad33bf2ea40b3c9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:38:07.955448 6469 ts_tablet_manager.cc:1905] T 8c7ecedde73b472084bd9925f2b76d9f P 4e95ca28b5c046e3ad33bf2ea40b3c9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:38:07.955854 6468 ts_tablet_manager.cc:1905] T 8c7ecedde73b472084bd9925f2b76d9f P 5de3b8f51b5f4c309c79028c23043efb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:38:07.958580 6469 ts_tablet_manager.cc:1918] T 8c7ecedde73b472084bd9925f2b76d9f P 4e95ca28b5c046e3ad33bf2ea40b3c9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:38:07.958768 6469 log.cc:1196] T 8c7ecedde73b472084bd9925f2b76d9f P 4e95ca28b5c046e3ad33bf2ea40b3c9e: Deleting WAL directory at /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786401215402979-17317-0/minicluster-data/ts-1/wal/wals/8c7ecedde73b472084bd9925f2b76d9f
I20260810 22:38:07.959148 6469 ts_tablet_manager.cc:1939] T 8c7ecedde73b472084bd9925f2b76d9f P 4e95ca28b5c046e3ad33bf2ea40b3c9e: Deleting consensus metadata
I20260810 22:38:07.959684 6468 ts_tablet_manager.cc:1918] T 8c7ecedde73b472084bd9925f2b76d9f P 5de3b8f51b5f4c309c79028c23043efb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:38:07.959856 6468 log.cc:1196] T 8c7ecedde73b472084bd9925f2b76d9f P 5de3b8f51b5f4c309c79028c23043efb: Deleting WAL directory at /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786401215402979-17317-0/minicluster-data/ts-2/wal/wals/8c7ecedde73b472084bd9925f2b76d9f
I20260810 22:38:07.960263 6468 ts_tablet_manager.cc:1939] T 8c7ecedde73b472084bd9925f2b76d9f P 5de3b8f51b5f4c309c79028c23043efb: Deleting consensus metadata
I20260810 22:38:07.960933 5915 catalog_manager.cc:4841] TS 4e95ca28b5c046e3ad33bf2ea40b3c9e (127.16.233.66:36463): tablet 8c7ecedde73b472084bd9925f2b76d9f (table test-table [id=04b847cfa420493c8f9fe0dd484b5669]) successfully deleted
I20260810 22:38:07.961599 5913 catalog_manager.cc:4841] TS 5de3b8f51b5f4c309c79028c23043efb (127.16.233.67:45363): tablet 8c7ecedde73b472084bd9925f2b76d9f (table test-table [id=04b847cfa420493c8f9fe0dd484b5669]) successfully deleted
W20260810 22:38:07.962800 5913 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.233.65:33947 (error 108)
I20260810 22:38:07.963792 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 6134
I20260810 22:38:07.964027 6437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:07.963652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.65:33947 (local address 127.0.0.1:38448)
0810 22:38:07.963823 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:07.963917 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.65:33947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260810 22:38:07.965612 5913 catalog_manager.cc:4584] TS 2babc051f64045f1a8fda2f9ec65e24e (127.16.233.65:33947): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8c7ecedde73b472084bd9925f2b76d9f: Network error: Client connection negotiation failed: client connection to 127.16.233.65:33947: connect: Connection refused (error 111)
I20260810 22:38:07.971876 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 6281
I20260810 22:38:07.980638 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 5897
2026-08-10T22:38:07Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1364 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:38:08 dist-test-slave-77v9 krb5kdc[6473](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:38:08 dist-test-slave-77v9 krb5kdc[6473](info): set up 2 sockets
Aug 10 22:38:08 dist-test-slave-77v9 krb5kdc[6473](info): commencing operation
krb5kdc: starting...
W20260810 22:38:09.174548 17317 mini_kdc.cc:121] Time spent starting KDC: real 1.169s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Aug 10 22:38:09 dist-test-slave-77v9 krb5kdc[6473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:38:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:38:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.233.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab.
Entry for principal kudu/127.16.233.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab.
I20260810 22:38:09.320302 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:36361
--webserver_interface=127.16.233.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab
--principal=kudu/127.16.233.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:38961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.126:36361
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:38:09.446238 6486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:38:09.446573 6486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:38:09.446617 6486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:38:09.450865 6486 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:38:09.450950 6486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:38:09.450973 6486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:38:09.450990 6486 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:38:09.451006 6486 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:38:09.456640 6486 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:38961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.126:36361
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab
--principal=kudu/127.16.233.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:36361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6486
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:38:09.457831 6486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:38:09.458920 6486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:38:09.469312 6493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:38:09.469506 6486 server_base.cc:1034] running on GCE node
W20260810 22:38:09.469329 6492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:09.469329 6495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:38:09.470427 6486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:38:09.471925 6486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:38:09.473140 6486 hybrid_clock.cc:648] HybridClock initialized: now 1786401489473095 us; error 58 us; skew 500 ppm
Aug 10 22:38:09 dist-test-slave-77v9 krb5kdc[6473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401489, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:38:09.476265 6486 init.cc:377] Logged in from keytab as kudu/127.16.233.126@KRBTEST.COM (short username kudu)
I20260810 22:38:09.477419 6486 webserver.cc:460] Webserver started at http://127.16.233.126:46747/ using document root <none> and password file <none>
I20260810 22:38:09.478093 6486 fs_manager.cc:359] Metadata directory not provided
I20260810 22:38:09.478200 6486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:38:09.478456 6486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:38:09.480731 6486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/minicluster-data/master-0/data/instance:
uuid: "13c2ab44d7094edc8b2c1d387f87d789"
format_stamp: "Formatted at 2026-08-10 22:38:09 on dist-test-slave-77v9"
server_key: "071b8a07d3369c9d66c5931c5c508f6f"
server_key_iv: "9967e8acbb5c0556f5608f5a0e1e6e7d"
server_key_version: "encryptionkey@0"
I20260810 22:38:09.481426 6486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/minicluster-data/master-0/wal/instance:
uuid: "13c2ab44d7094edc8b2c1d387f87d789"
format_stamp: "Formatted at 2026-08-10 22:38:09 on dist-test-slave-77v9"
server_key: "071b8a07d3369c9d66c5931c5c508f6f"
server_key_iv: "9967e8acbb5c0556f5608f5a0e1e6e7d"
server_key_version: "encryptionkey@0"
I20260810 22:38:09.485617 6486 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 22:38:09.488323 6503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:09.489836 6486 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:38:09.490052 6486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/minicluster-data/master-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/minicluster-data/master-0/wal
uuid: "13c2ab44d7094edc8b2c1d387f87d789"
format_stamp: "Formatted at 2026-08-10 22:38:09 on dist-test-slave-77v9"
server_key: "071b8a07d3369c9d66c5931c5c508f6f"
server_key_iv: "9967e8acbb5c0556f5608f5a0e1e6e7d"
server_key_version: "encryptionkey@0"
I20260810 22:38:09.490196 6486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:38:09.511467 6486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:38:09.515259 6486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:38:09.515518 6486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:38:09.524072 6486 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.126:36361
I20260810 22:38:09.524096 6560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.126:36361 every 8 connection(s)
I20260810 22:38:09.525218 6486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
I20260810 22:38:09.528286 6561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:38:09.529934 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 6486
I20260810 22:38:09.530102 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/minicluster-data/master-0/wal/instance
I20260810 22:38:09.530441 17317 external_mini_cluster.cc:1428] Setting key 2d31a02df91cb6b74cefb936767aa545
I20260810 22:38:09.535482 6561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13c2ab44d7094edc8b2c1d387f87d789: Bootstrap starting.
I20260810 22:38:09.538354 6561 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13c2ab44d7094edc8b2c1d387f87d789: Neither blocks nor log segments found. Creating new log.
I20260810 22:38:09.539152 6561 log.cc:824] T 00000000000000000000000000000000 P 13c2ab44d7094edc8b2c1d387f87d789: Log is configured to *not* fsync() on all Append() calls
Aug 10 22:38:09 dist-test-slave-77v9 krb5kdc[6473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:38:09.541505 6561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13c2ab44d7094edc8b2c1d387f87d789: No bootstrap required, opened a new log
I20260810 22:38:09.545338 6561 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13c2ab44d7094edc8b2c1d387f87d789 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13c2ab44d7094edc8b2c1d387f87d789" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 36361 } }
I20260810 22:38:09.545642 6561 raft_consensus.cc:374] T 00000000000000000000000000000000 P 13c2ab44d7094edc8b2c1d387f87d789 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:38:09.545724 6561 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13c2ab44d7094edc8b2c1d387f87d789 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13c2ab44d7094edc8b2c1d387f87d789, State: Initialized, Role: FOLLOWER
I20260810 22:38:09.546211 6561 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13c2ab44d7094edc8b2c1d387f87d789 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13c2ab44d7094edc8b2c1d387f87d789" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 36361 } }
I20260810 22:38:09.546350 6561 raft_consensus.cc:388] T 00000000000000000000000000000000 P 13c2ab44d7094edc8b2c1d387f87d789 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:38:09.546403 6564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:09.532044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47424 (local address 127.16.233.126:36361)
0810 22:38:09.532507 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:09.532525 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:09.532563 (+ 38us) server_negotiation.cc:407] Connection header received
0810 22:38:09.533310 (+ 747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:09.533330 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:09.533666 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:09.533966 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:09.535012 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:09.536120 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:09.537344 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:09.537835 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:09.537968 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:09.541076 (+ 3108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:09.541081 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:09.541096 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:38:09.541126 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:09.543191 (+ 2065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:09.543712 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:09.543719 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:09.543723 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:09.543847 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:09.544162 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:09.544166 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:09.544168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:09.544590 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:09.544888 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:09.545150 (+ 262us) server_negotiation.cc:299] Negotiation successful
0810 22:38:09.545461 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":151,"threads_started":1}
I20260810 22:38:09.546440 6561 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13c2ab44d7094edc8b2c1d387f87d789 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:38:09.546641 6561 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13c2ab44d7094edc8b2c1d387f87d789 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:38:09.547871 6561 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13c2ab44d7094edc8b2c1d387f87d789 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13c2ab44d7094edc8b2c1d387f87d789" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 36361 } }
I20260810 22:38:09.548259 6561 leader_election.cc:302] T 00000000000000000000000000000000 P 13c2ab44d7094edc8b2c1d387f87d789 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13c2ab44d7094edc8b2c1d387f87d789; no voters:
I20260810 22:38:09.548635 6561 leader_election.cc:288] T 00000000000000000000000000000000 P 13c2ab44d7094edc8b2c1d387f87d789 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:38:09.548851 6567 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13c2ab44d7094edc8b2c1d387f87d789 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:38:09.549170 6567 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13c2ab44d7094edc8b2c1d387f87d789 [term 1 LEADER]: Becoming Leader. State: Replica: 13c2ab44d7094edc8b2c1d387f87d789, State: Running, Role: LEADER
I20260810 22:38:09.549525 6567 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13c2ab44d7094edc8b2c1d387f87d789 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13c2ab44d7094edc8b2c1d387f87d789" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 36361 } }
I20260810 22:38:09.549986 6561 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13c2ab44d7094edc8b2c1d387f87d789 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:38:09.551433 6568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13c2ab44d7094edc8b2c1d387f87d789 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "13c2ab44d7094edc8b2c1d387f87d789" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13c2ab44d7094edc8b2c1d387f87d789" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 36361 } } }
I20260810 22:38:09.551564 6568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13c2ab44d7094edc8b2c1d387f87d789 [sys.catalog]: This master's current role is: LEADER
I20260810 22:38:09.551499 6569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13c2ab44d7094edc8b2c1d387f87d789 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13c2ab44d7094edc8b2c1d387f87d789. Latest consensus state: current_term: 1 leader_uuid: "13c2ab44d7094edc8b2c1d387f87d789" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13c2ab44d7094edc8b2c1d387f87d789" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 36361 } } }
I20260810 22:38:09.551615 6569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13c2ab44d7094edc8b2c1d387f87d789 [sys.catalog]: This master's current role is: LEADER
I20260810 22:38:09.551947 6578 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:38:09.555675 6578 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:38:09.562263 6578 catalog_manager.cc:1347] Generated new cluster ID: 34a39d8013254974b73ecd4e21cede37
I20260810 22:38:09.562368 6578 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:38:09.606534 6578 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:38:09.607957 6578 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:38:09.615012 6578 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 13c2ab44d7094edc8b2c1d387f87d789: Generated new TSK 0
I20260810 22:38:09.615840 6578 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.233.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab.
Entry for principal kudu/127.16.233.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab.
I20260810 22:38:09.653554 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.65:0
--local_ip_for_outbound_sockets=127.16.233.65
--webserver_interface=127.16.233.65
--webserver_port=0
--tserver_master_addrs=127.16.233.126:36361
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab
--principal=kudu/127.16.233.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:38961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:38:09.781934 6589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:38:09.782325 6589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:38:09.782414 6589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:38:09.786852 6589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:38:09.786980 6589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:38:09.787083 6589 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.65
I20260810 22:38:09.793087 6589 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:38961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.65.keytab
--principal=kudu/127.16.233.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.233.65
--webserver_port=0
--tserver_master_addrs=127.16.233.126:36361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.65
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:38:09.794476 6589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:38:09.795703 6589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:38:09.806401 6597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:09.806401 6595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:09.806486 6594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:38:09.807021 6589 server_base.cc:1034] running on GCE node
I20260810 22:38:09.807521 6589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:38:09.808314 6589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:38:09.809528 6589 hybrid_clock.cc:648] HybridClock initialized: now 1786401489809513 us; error 27 us; skew 500 ppm
Aug 10 22:38:09 dist-test-slave-77v9 krb5kdc[6473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401489, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:38:09.812649 6589 init.cc:377] Logged in from keytab as kudu/127.16.233.65@KRBTEST.COM (short username kudu)
I20260810 22:38:09.813694 6589 webserver.cc:460] Webserver started at http://127.16.233.65:39693/ using document root <none> and password file <none>
I20260810 22:38:09.814318 6589 fs_manager.cc:359] Metadata directory not provided
I20260810 22:38:09.814391 6589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:38:09.814626 6589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:38:09.817094 6589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/minicluster-data/ts-0/data/instance:
uuid: "2efcb37f49184c2f9e10cb891c522ea2"
format_stamp: "Formatted at 2026-08-10 22:38:09 on dist-test-slave-77v9"
server_key: "bf952db57332612a83812326701755b6"
server_key_iv: "ddfffa7174afd37c47e46ef6cd2d9fd5"
server_key_version: "encryptionkey@0"
I20260810 22:38:09.817766 6589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/minicluster-data/ts-0/wal/instance:
uuid: "2efcb37f49184c2f9e10cb891c522ea2"
format_stamp: "Formatted at 2026-08-10 22:38:09 on dist-test-slave-77v9"
server_key: "bf952db57332612a83812326701755b6"
server_key_iv: "ddfffa7174afd37c47e46ef6cd2d9fd5"
server_key_version: "encryptionkey@0"
I20260810 22:38:09.821928 6589 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 22:38:09.824652 6604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:09.826083 6589 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 22:38:09.826229 6589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/minicluster-data/ts-0/wal
uuid: "2efcb37f49184c2f9e10cb891c522ea2"
format_stamp: "Formatted at 2026-08-10 22:38:09 on dist-test-slave-77v9"
server_key: "bf952db57332612a83812326701755b6"
server_key_iv: "ddfffa7174afd37c47e46ef6cd2d9fd5"
server_key_version: "encryptionkey@0"
I20260810 22:38:09.826351 6589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:38:09.843732 6589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:38:09.847162 6589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:38:09.847399 6589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:38:09.848063 6589 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:38:09.849059 6589 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:38:09.849112 6589 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:09.849182 6589 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:38:09.849215 6589 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:09.859575 6589 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.65:40817
I20260810 22:38:09.859601 6727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.65:40817 every 8 connection(s)
I20260810 22:38:09.860762 6589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
I20260810 22:38:09.863893 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 6589
I20260810 22:38:09.864053 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/minicluster-data/ts-0/wal/instance
I20260810 22:38:09.864396 17317 external_mini_cluster.cc:1428] Setting key 95bf079f59184b00a9ab090c5a3d7f9c
Aug 10 22:38:09 dist-test-slave-77v9 krb5kdc[6473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401489, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.65@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.233.66@KRBTEST.COM; defaulting to no policy
I20260810 22:38:09.876911 6564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:09.862878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:38443 (local address 127.16.233.126:36361)
0810 22:38:09.863096 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:09.863102 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:09.864004 (+ 902us) server_negotiation.cc:407] Connection header received
0810 22:38:09.865649 (+ 1645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:09.865653 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:09.865708 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:09.865797 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:09.867062 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:09.867611 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:09.868835 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:09.869096 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:09.869216 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:09.872362 (+ 3146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:09.872369 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:09.872376 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:38:09.872423 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:09.874291 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:09.874925 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:09.874932 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:09.874936 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:09.875041 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:09.875417 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:09.875421 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:09.875423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:09.875700 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:09.875823 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:09.876485 (+ 662us) server_negotiation.cc:299] Negotiation successful
0810 22:38:09.876650 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260810 22:38:09.877995 6730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:09.863291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:36361 (local address 127.16.233.65:38443)
0810 22:38:09.863791 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:09.863845 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 22:38:09.865435 (+ 1590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:38:09.866009 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:38:09.866017 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:38:09.866382 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:38:09.866904 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:09.866918 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:09.867754 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:09.867758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:09.868684 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:09.868694 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:09.869265 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:09.869270 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:09.869515 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:09.870266 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:38:09.870292 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:38:09.872158 (+ 1866us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:38:09.874488 (+ 2330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:38:09.874493 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:38:09.874508 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:38:09.874805 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:38:09.875146 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:38:09.875148 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:38:09.875149 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:38:09.875302 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:38:09.875882 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:38:09.875888 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:38:09.876184 (+ 296us) client_negotiation.cc:769] Sending connection context
0810 22:38:09.876385 (+ 201us) client_negotiation.cc:240] Negotiation successful
0810 22:38:09.876929 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":142,"threads_started":1}
I20260810 22:38:09.879671 6728 heartbeater.cc:347] Connected to a master server at 127.16.233.126:36361
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.66@KRBTEST.COM" created.
I20260810 22:38:09.879997 6728 heartbeater.cc:464] Registering TS with master...
I20260810 22:38:09.880641 6728 heartbeater.cc:511] Master 127.16.233.126:36361 requested a full tablet report, sending...
I20260810 22:38:09.882649 6521 ts_manager.cc:194] Registered new tserver with Master: 2efcb37f49184c2f9e10cb891c522ea2 (127.16.233.65:40817)
I20260810 22:38:09.884348 6521 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.65@KRBTEST.COM'} at 127.16.233.65:38443
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab.
Entry for principal kudu/127.16.233.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab.
I20260810 22:38:09.901535 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.66:0
--local_ip_for_outbound_sockets=127.16.233.66
--webserver_interface=127.16.233.66
--webserver_port=0
--tserver_master_addrs=127.16.233.126:36361
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab
--principal=kudu/127.16.233.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:38961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:38:10.020521 6734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:38:10.020859 6734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:38:10.020915 6734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:38:10.025295 6734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:38:10.025378 6734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:38:10.025493 6734 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.66
I20260810 22:38:10.031519 6734 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:38961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.66.keytab
--principal=kudu/127.16.233.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.233.66
--webserver_port=0
--tserver_master_addrs=127.16.233.126:36361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.66
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:38:10.032766 6734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:38:10.033924 6734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:38:10.044512 6740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:10.044503 6742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:10.044503 6739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:38:10.044991 6734 server_base.cc:1034] running on GCE node
I20260810 22:38:10.045467 6734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:38:10.046242 6734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:38:10.047474 6734 hybrid_clock.cc:648] HybridClock initialized: now 1786401490047433 us; error 58 us; skew 500 ppm
Aug 10 22:38:10 dist-test-slave-77v9 krb5kdc[6473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401490, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:38:10.051082 6734 init.cc:377] Logged in from keytab as kudu/127.16.233.66@KRBTEST.COM (short username kudu)
I20260810 22:38:10.052456 6734 webserver.cc:460] Webserver started at http://127.16.233.66:38011/ using document root <none> and password file <none>
I20260810 22:38:10.053174 6734 fs_manager.cc:359] Metadata directory not provided
I20260810 22:38:10.053267 6734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:38:10.053480 6734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:38:10.055927 6734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/minicluster-data/ts-1/data/instance:
uuid: "04f18a0f77be4c61b338009521fda0d8"
format_stamp: "Formatted at 2026-08-10 22:38:10 on dist-test-slave-77v9"
server_key: "37aa2e2deebbf4230ed70cbdf23bf736"
server_key_iv: "b6a87eb516c6d82664c35a094e1179d4"
server_key_version: "encryptionkey@0"
I20260810 22:38:10.056720 6734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/minicluster-data/ts-1/wal/instance:
uuid: "04f18a0f77be4c61b338009521fda0d8"
format_stamp: "Formatted at 2026-08-10 22:38:10 on dist-test-slave-77v9"
server_key: "37aa2e2deebbf4230ed70cbdf23bf736"
server_key_iv: "b6a87eb516c6d82664c35a094e1179d4"
server_key_version: "encryptionkey@0"
I20260810 22:38:10.061424 6734 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 22:38:10.064361 6749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:10.066020 6734 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:38:10.066195 6734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/minicluster-data/ts-1/wal
uuid: "04f18a0f77be4c61b338009521fda0d8"
format_stamp: "Formatted at 2026-08-10 22:38:10 on dist-test-slave-77v9"
server_key: "37aa2e2deebbf4230ed70cbdf23bf736"
server_key_iv: "b6a87eb516c6d82664c35a094e1179d4"
server_key_version: "encryptionkey@0"
I20260810 22:38:10.066355 6734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:38:10.084304 6734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:38:10.087935 6734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:38:10.088193 6734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:38:10.088979 6734 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:38:10.090004 6734 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:38:10.090060 6734 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:10.090106 6734 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:38:10.090123 6734 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:10.100781 6734 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.66:36535
I20260810 22:38:10.100862 6871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.66:36535 every 8 connection(s)
I20260810 22:38:10.101910 6734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
I20260810 22:38:10.111203 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 6734
I20260810 22:38:10.111385 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/minicluster-data/ts-1/wal/instance
I20260810 22:38:10.111726 17317 external_mini_cluster.cc:1428] Setting key 1d800407c491de0924fd2697d811dd1c
Aug 10 22:38:10 dist-test-slave-77v9 krb5kdc[6473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401490, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.66@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:38:10.119938 6564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:10.104241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:53555 (local address 127.16.233.126:36361)
0810 22:38:10.104459 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:10.104463 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:10.105579 (+ 1116us) server_negotiation.cc:407] Connection header received
0810 22:38:10.107354 (+ 1775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:10.107358 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:10.107442 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:10.107536 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:10.109053 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:10.109671 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:10.111357 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:10.111839 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:10.111989 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:10.115438 (+ 3449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:10.115442 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:10.115444 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:38:10.115473 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:10.117359 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:10.118093 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:10.118097 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:10.118099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:10.118189 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:10.118616 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:10.118619 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:10.118622 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:10.118844 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:10.118969 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:10.119596 (+ 627us) server_negotiation.cc:299] Negotiation successful
0810 22:38:10.119751 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260810 22:38:10.121021 6874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:10.104722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:36361 (local address 127.16.233.66:53555)
0810 22:38:10.105370 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:10.105434 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0810 22:38:10.107054 (+ 1620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:38:10.107753 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:38:10.107761 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:38:10.108213 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:38:10.108800 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:10.108859 (+ 59us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:10.109840 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:10.109882 (+ 42us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:10.111125 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:10.111140 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:10.112064 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:10.112069 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:10.112298 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:10.113063 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:38:10.113083 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:38:10.115193 (+ 2110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:38:10.117527 (+ 2334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:38:10.117535 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:38:10.117560 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:38:10.117943 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:38:10.118297 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:38:10.118304 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:38:10.118309 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:38:10.118493 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:38:10.118970 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:38:10.118979 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:38:10.119315 (+ 336us) client_negotiation.cc:769] Sending connection context
0810 22:38:10.119528 (+ 213us) client_negotiation.cc:240] Negotiation successful
0810 22:38:10.119942 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":161,"threads_started":1}
I20260810 22:38:10.122493 6872 heartbeater.cc:347] Connected to a master server at 127.16.233.126:36361
I20260810 22:38:10.122809 6872 heartbeater.cc:464] Registering TS with master...
I20260810 22:38:10.123413 6872 heartbeater.cc:511] Master 127.16.233.126:36361 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.16.233.67@KRBTEST.COM; defaulting to no policy
I20260810 22:38:10.124758 6521 ts_manager.cc:194] Registered new tserver with Master: 04f18a0f77be4c61b338009521fda0d8 (127.16.233.66:36535)
I20260810 22:38:10.125654 6521 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.66@KRBTEST.COM'} at 127.16.233.66:53555
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab.
Entry for principal kudu/127.16.233.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab.
I20260810 22:38:10.149832 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.67:0
--local_ip_for_outbound_sockets=127.16.233.67
--webserver_interface=127.16.233.67
--webserver_port=0
--tserver_master_addrs=127.16.233.126:36361
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab
--principal=kudu/127.16.233.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:38961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:38:10.271695 6878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:38:10.272047 6878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:38:10.272094 6878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:38:10.276095 6878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:38:10.276156 6878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:38:10.276213 6878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.67
I20260810 22:38:10.281678 6878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:38961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.67.keytab
--principal=kudu/127.16.233.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.233.67
--webserver_port=0
--tserver_master_addrs=127.16.233.126:36361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.67
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:38:10.282790 6878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:38:10.283815 6878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:38:10.294057 6883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:38:10.294123 6878 server_base.cc:1034] running on GCE node
W20260810 22:38:10.294054 6884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:10.294050 6886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:38:10.294703 6878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:38:10.295434 6878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:38:10.296610 6878 hybrid_clock.cc:648] HybridClock initialized: now 1786401490296590 us; error 33 us; skew 500 ppm
Aug 10 22:38:10 dist-test-slave-77v9 krb5kdc[6473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401490, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:38:10.306838 6878 init.cc:377] Logged in from keytab as kudu/127.16.233.67@KRBTEST.COM (short username kudu)
I20260810 22:38:10.308003 6878 webserver.cc:460] Webserver started at http://127.16.233.67:45227/ using document root <none> and password file <none>
I20260810 22:38:10.308663 6878 fs_manager.cc:359] Metadata directory not provided
I20260810 22:38:10.308780 6878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:38:10.309105 6878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:38:10.311424 6878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/minicluster-data/ts-2/data/instance:
uuid: "84023a2618eb412598d0d0d07a70dbbc"
format_stamp: "Formatted at 2026-08-10 22:38:10 on dist-test-slave-77v9"
server_key: "487c85eecee046c7f91913fa864a062b"
server_key_iv: "11fb7eb76e2f4cebb1a5943734bdc8bc"
server_key_version: "encryptionkey@0"
I20260810 22:38:10.312067 6878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/minicluster-data/ts-2/wal/instance:
uuid: "84023a2618eb412598d0d0d07a70dbbc"
format_stamp: "Formatted at 2026-08-10 22:38:10 on dist-test-slave-77v9"
server_key: "487c85eecee046c7f91913fa864a062b"
server_key_iv: "11fb7eb76e2f4cebb1a5943734bdc8bc"
server_key_version: "encryptionkey@0"
I20260810 22:38:10.316310 6878 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 22:38:10.319243 6893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:10.320719 6878 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 22:38:10.321012 6878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/minicluster-data/ts-2/wal
uuid: "84023a2618eb412598d0d0d07a70dbbc"
format_stamp: "Formatted at 2026-08-10 22:38:10 on dist-test-slave-77v9"
server_key: "487c85eecee046c7f91913fa864a062b"
server_key_iv: "11fb7eb76e2f4cebb1a5943734bdc8bc"
server_key_version: "encryptionkey@0"
I20260810 22:38:10.321151 6878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:38:10.342499 6878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:38:10.346276 6878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:38:10.346521 6878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:38:10.347213 6878 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:38:10.348122 6878 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:38:10.348198 6878 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:10.348281 6878 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:38:10.348326 6878 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:10.358100 6878 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.67:45187
I20260810 22:38:10.358127 7013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.67:45187 every 8 connection(s)
I20260810 22:38:10.359105 6878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
I20260810 22:38:10.369318 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 6878
I20260810 22:38:10.369513 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/minicluster-data/ts-2/wal/instance
I20260810 22:38:10.369817 17317 external_mini_cluster.cc:1428] Setting key 6256afc4e4ca6cedd33339d0ac602c01
Aug 10 22:38:10 dist-test-slave-77v9 krb5kdc[6473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401490, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.67@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:38:10.375237 6564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:10.361081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:45343 (local address 127.16.233.126:36361)
0810 22:38:10.361250 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:10.361255 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:10.362151 (+ 896us) server_negotiation.cc:407] Connection header received
0810 22:38:10.364202 (+ 2051us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:10.364205 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:10.364255 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:10.364334 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:10.366078 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:10.366626 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:10.367978 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:10.368195 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:10.368273 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:10.371500 (+ 3227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:10.371504 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:10.371507 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:38:10.371543 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:10.373226 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:10.373748 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:10.373750 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:10.373751 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:10.373827 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:10.374182 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:10.374184 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:10.374185 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:10.374370 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:10.374448 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:10.375001 (+ 553us) server_negotiation.cc:299] Negotiation successful
0810 22:38:10.375113 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 22:38:10.376153 7016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:10.361413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:36361 (local address 127.16.233.67:45343)
0810 22:38:10.361908 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:10.362004 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0810 22:38:10.363961 (+ 1957us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:38:10.364553 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:38:10.364559 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:38:10.365066 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:38:10.365894 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:10.365913 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:10.366812 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:10.366815 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:10.367834 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:10.367844 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:10.368344 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:10.368347 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:10.368506 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:10.369202 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:38:10.369222 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:38:10.371328 (+ 2106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:38:10.373336 (+ 2008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:38:10.373342 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:38:10.373356 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:38:10.373648 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:38:10.373929 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:38:10.373932 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:38:10.373934 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:38:10.374086 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:38:10.374463 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:38:10.374468 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:38:10.374727 (+ 259us) client_negotiation.cc:769] Sending connection context
0810 22:38:10.374886 (+ 159us) client_negotiation.cc:240] Negotiation successful
0810 22:38:10.375203 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":122,"threads_started":1}
I20260810 22:38:10.377475 7014 heartbeater.cc:347] Connected to a master server at 127.16.233.126:36361
I20260810 22:38:10.377746 7014 heartbeater.cc:464] Registering TS with master...
I20260810 22:38:10.378291 7014 heartbeater.cc:511] Master 127.16.233.126:36361 requested a full tablet report, sending...
I20260810 22:38:10.379467 6521 ts_manager.cc:194] Registered new tserver with Master: 84023a2618eb412598d0d0d07a70dbbc (127.16.233.67:45187)
I20260810 22:38:10.380097 6521 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.233.67@KRBTEST.COM'} at 127.16.233.67:45343
I20260810 22:38:10.384179 17317 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 22:38:10.395267 6564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:10.386355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47434 (local address 127.16.233.126:36361)
0810 22:38:10.386512 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:10.386517 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:10.386711 (+ 194us) server_negotiation.cc:407] Connection header received
0810 22:38:10.386815 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:10.386817 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:10.386858 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:10.386932 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:10.387692 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:10.388212 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:10.389244 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:10.389465 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:10.389549 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:10.391422 (+ 1873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:10.391425 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:10.391427 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:38:10.391453 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:10.393118 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:10.393653 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:10.393659 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:10.393661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:10.393764 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:10.394105 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:10.394114 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:10.394115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:10.394379 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:10.394549 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:10.394910 (+ 361us) server_negotiation.cc:299] Negotiation successful
0810 22:38:10.395046 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 22:38:10.398970 6521 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47434:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 22:38:10.402321 6521 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:38:10.418655 7025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:10.412551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.67:45187 (local address 127.0.0.1:36950)
0810 22:38:10.413031 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:10.413090 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 22:38:10.413231 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:38:10.414387 (+ 1156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:38:10.414392 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:38:10.414416 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:38:10.414626 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:10.414635 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:10.415805 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:10.415811 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:10.417009 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:10.417020 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:10.418120 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:10.418124 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:10.418234 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:10.418274 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 22:38:10.418380 (+ 106us) client_negotiation.cc:240] Negotiation successful
0810 22:38:10.418508 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":110,"threads_started":1}
I20260810 22:38:10.418665 7028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:10.413340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36950 (local address 127.16.233.67:45187)
0810 22:38:10.413836 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:10.413852 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:10.413879 (+ 27us) server_negotiation.cc:407] Connection header received
0810 22:38:10.413965 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:10.413971 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:10.414199 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:10.414381 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:38:10.414778 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:10.415685 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:10.417152 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:10.417992 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:10.418118 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:10.418365 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:10.418420 (+ 55us) server_negotiation.cc:299] Negotiation successful
0810 22:38:10.418511 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":165,"threads_started":1}
I20260810 22:38:10.419437 7027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:10.412841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.66:36535 (local address 127.0.0.1:48150)
0810 22:38:10.413816 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:10.413831 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:38:10.413929 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:38:10.414732 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:38:10.414734 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:38:10.414753 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:38:10.414857 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:10.414863 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:10.416376 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:10.416380 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:10.417647 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:10.417658 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:10.419028 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:10.419032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:10.419118 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:10.419141 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 22:38:10.419221 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 22:38:10.419302 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":894,"mutex_wait_us":61,"thread_start_us":131,"threads_started":1}
I20260810 22:38:10.419783 7030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:10.413469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48150 (local address 127.16.233.66:36535)
0810 22:38:10.414284 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:10.414300 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:10.414324 (+ 24us) server_negotiation.cc:407] Connection header received
0810 22:38:10.414400 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:10.414405 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:10.414583 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:10.414756 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:38:10.414988 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:10.416212 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:10.417795 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:10.418891 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:10.419016 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:10.419146 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:10.419485 (+ 339us) server_negotiation.cc:299] Negotiation successful
0810 22:38:10.419591 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":159,"threads_started":1}
I20260810 22:38:10.420403 7026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:10.413010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.65:40817 (local address 127.0.0.1:40718)
0810 22:38:10.413845 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:10.413861 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 22:38:10.413934 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:38:10.415547 (+ 1613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:38:10.415550 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:38:10.415569 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:38:10.415652 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:10.415658 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:10.417011 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:10.417019 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:10.418517 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:10.418530 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:10.419940 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:10.419944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:10.420045 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:10.420069 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 22:38:10.420183 (+ 114us) client_negotiation.cc:240] Negotiation successful
0810 22:38:10.420270 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":785,"mutex_wait_us":6,"thread_start_us":105,"threads_started":1}
I20260810 22:38:10.420866 7029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:10.413457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40718 (local address 127.16.233.65:40817)
0810 22:38:10.415071 (+ 1614us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:10.415078 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:10.415101 (+ 23us) server_negotiation.cc:407] Connection header received
0810 22:38:10.415177 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:10.415186 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:10.415382 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:10.415528 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:38:10.415784 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:10.416862 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:10.418679 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:10.419608 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:10.419715 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:10.419856 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:10.420591 (+ 735us) server_negotiation.cc:299] Negotiation successful
0810 22:38:10.420695 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1496,"thread_start_us":85,"threads_started":1}
I20260810 22:38:10.422612 6802 tablet_service.cc:1467] Processing CreateTablet for tablet fb4f0ea982684347aef0a8a1dc29c4ca (DEFAULT_TABLE table=test-table [id=6c1871b173014c27abd6a51d3dd7ed08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:38:10.422663 6942 tablet_service.cc:1467] Processing CreateTablet for tablet fb4f0ea982684347aef0a8a1dc29c4ca (DEFAULT_TABLE table=test-table [id=6c1871b173014c27abd6a51d3dd7ed08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:38:10.423683 6802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb4f0ea982684347aef0a8a1dc29c4ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:38:10.423799 6942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb4f0ea982684347aef0a8a1dc29c4ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:38:10.423775 6657 tablet_service.cc:1467] Processing CreateTablet for tablet fb4f0ea982684347aef0a8a1dc29c4ca (DEFAULT_TABLE table=test-table [id=6c1871b173014c27abd6a51d3dd7ed08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:38:10.424786 6657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb4f0ea982684347aef0a8a1dc29c4ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:38:10.430019 7031 tablet_bootstrap.cc:492] T fb4f0ea982684347aef0a8a1dc29c4ca P 04f18a0f77be4c61b338009521fda0d8: Bootstrap starting.
I20260810 22:38:10.431537 7032 tablet_bootstrap.cc:492] T fb4f0ea982684347aef0a8a1dc29c4ca P 84023a2618eb412598d0d0d07a70dbbc: Bootstrap starting.
I20260810 22:38:10.432754 7031 tablet_bootstrap.cc:654] T fb4f0ea982684347aef0a8a1dc29c4ca P 04f18a0f77be4c61b338009521fda0d8: Neither blocks nor log segments found. Creating new log.
I20260810 22:38:10.433059 7033 tablet_bootstrap.cc:492] T fb4f0ea982684347aef0a8a1dc29c4ca P 2efcb37f49184c2f9e10cb891c522ea2: Bootstrap starting.
I20260810 22:38:10.433842 7032 tablet_bootstrap.cc:654] T fb4f0ea982684347aef0a8a1dc29c4ca P 84023a2618eb412598d0d0d07a70dbbc: Neither blocks nor log segments found. Creating new log.
I20260810 22:38:10.434242 7031 log.cc:824] T fb4f0ea982684347aef0a8a1dc29c4ca P 04f18a0f77be4c61b338009521fda0d8: Log is configured to *not* fsync() on all Append() calls
I20260810 22:38:10.435038 7032 log.cc:824] T fb4f0ea982684347aef0a8a1dc29c4ca P 84023a2618eb412598d0d0d07a70dbbc: Log is configured to *not* fsync() on all Append() calls
I20260810 22:38:10.435068 7033 tablet_bootstrap.cc:654] T fb4f0ea982684347aef0a8a1dc29c4ca P 2efcb37f49184c2f9e10cb891c522ea2: Neither blocks nor log segments found. Creating new log.
I20260810 22:38:10.435902 7033 log.cc:824] T fb4f0ea982684347aef0a8a1dc29c4ca P 2efcb37f49184c2f9e10cb891c522ea2: Log is configured to *not* fsync() on all Append() calls
I20260810 22:38:10.436537 7031 tablet_bootstrap.cc:492] T fb4f0ea982684347aef0a8a1dc29c4ca P 04f18a0f77be4c61b338009521fda0d8: No bootstrap required, opened a new log
I20260810 22:38:10.436801 7031 ts_tablet_manager.cc:1397] T fb4f0ea982684347aef0a8a1dc29c4ca P 04f18a0f77be4c61b338009521fda0d8: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.004s
I20260810 22:38:10.437095 7032 tablet_bootstrap.cc:492] T fb4f0ea982684347aef0a8a1dc29c4ca P 84023a2618eb412598d0d0d07a70dbbc: No bootstrap required, opened a new log
I20260810 22:38:10.437362 7032 ts_tablet_manager.cc:1397] T fb4f0ea982684347aef0a8a1dc29c4ca P 84023a2618eb412598d0d0d07a70dbbc: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 22:38:10.437877 7033 tablet_bootstrap.cc:492] T fb4f0ea982684347aef0a8a1dc29c4ca P 2efcb37f49184c2f9e10cb891c522ea2: No bootstrap required, opened a new log
I20260810 22:38:10.438123 7033 ts_tablet_manager.cc:1397] T fb4f0ea982684347aef0a8a1dc29c4ca P 2efcb37f49184c2f9e10cb891c522ea2: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260810 22:38:10.441133 7031 raft_consensus.cc:348] T fb4f0ea982684347aef0a8a1dc29c4ca P 04f18a0f77be4c61b338009521fda0d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84023a2618eb412598d0d0d07a70dbbc" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 45187 } } peers { permanent_uuid: "04f18a0f77be4c61b338009521fda0d8" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 36535 } } peers { permanent_uuid: "2efcb37f49184c2f9e10cb891c522ea2" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 40817 } }
I20260810 22:38:10.441164 7032 raft_consensus.cc:348] T fb4f0ea982684347aef0a8a1dc29c4ca P 84023a2618eb412598d0d0d07a70dbbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84023a2618eb412598d0d0d07a70dbbc" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 45187 } } peers { permanent_uuid: "04f18a0f77be4c61b338009521fda0d8" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 36535 } } peers { permanent_uuid: "2efcb37f49184c2f9e10cb891c522ea2" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 40817 } }
I20260810 22:38:10.441443 7031 raft_consensus.cc:374] T fb4f0ea982684347aef0a8a1dc29c4ca P 04f18a0f77be4c61b338009521fda0d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:38:10.441443 7032 raft_consensus.cc:374] T fb4f0ea982684347aef0a8a1dc29c4ca P 84023a2618eb412598d0d0d07a70dbbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:38:10.441504 7032 raft_consensus.cc:729] T fb4f0ea982684347aef0a8a1dc29c4ca P 84023a2618eb412598d0d0d07a70dbbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84023a2618eb412598d0d0d07a70dbbc, State: Initialized, Role: FOLLOWER
I20260810 22:38:10.441535 7031 raft_consensus.cc:729] T fb4f0ea982684347aef0a8a1dc29c4ca P 04f18a0f77be4c61b338009521fda0d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04f18a0f77be4c61b338009521fda0d8, State: Initialized, Role: FOLLOWER
I20260810 22:38:10.441820 7033 raft_consensus.cc:348] T fb4f0ea982684347aef0a8a1dc29c4ca P 2efcb37f49184c2f9e10cb891c522ea2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84023a2618eb412598d0d0d07a70dbbc" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 45187 } } peers { permanent_uuid: "04f18a0f77be4c61b338009521fda0d8" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 36535 } } peers { permanent_uuid: "2efcb37f49184c2f9e10cb891c522ea2" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 40817 } }
I20260810 22:38:10.442091 7033 raft_consensus.cc:374] T fb4f0ea982684347aef0a8a1dc29c4ca P 2efcb37f49184c2f9e10cb891c522ea2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:38:10.442030 7031 consensus_queue.cc:260] T fb4f0ea982684347aef0a8a1dc29c4ca P 04f18a0f77be4c61b338009521fda0d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84023a2618eb412598d0d0d07a70dbbc" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 45187 } } peers { permanent_uuid: "04f18a0f77be4c61b338009521fda0d8" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 36535 } } peers { permanent_uuid: "2efcb37f49184c2f9e10cb891c522ea2" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 40817 } }
I20260810 22:38:10.442030 7032 consensus_queue.cc:260] T fb4f0ea982684347aef0a8a1dc29c4ca P 84023a2618eb412598d0d0d07a70dbbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84023a2618eb412598d0d0d07a70dbbc" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 45187 } } peers { permanent_uuid: "04f18a0f77be4c61b338009521fda0d8" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 36535 } } peers { permanent_uuid: "2efcb37f49184c2f9e10cb891c522ea2" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 40817 } }
I20260810 22:38:10.442167 7033 raft_consensus.cc:729] T fb4f0ea982684347aef0a8a1dc29c4ca P 2efcb37f49184c2f9e10cb891c522ea2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efcb37f49184c2f9e10cb891c522ea2, State: Initialized, Role: FOLLOWER
I20260810 22:38:10.442690 7033 consensus_queue.cc:260] T fb4f0ea982684347aef0a8a1dc29c4ca P 2efcb37f49184c2f9e10cb891c522ea2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84023a2618eb412598d0d0d07a70dbbc" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 45187 } } peers { permanent_uuid: "04f18a0f77be4c61b338009521fda0d8" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 36535 } } peers { permanent_uuid: "2efcb37f49184c2f9e10cb891c522ea2" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 40817 } }
I20260810 22:38:10.442800 6872 heartbeater.cc:503] Master 127.16.233.126:36361 was elected leader, sending a full tablet report...
I20260810 22:38:10.443105 7031 ts_tablet_manager.cc:1428] T fb4f0ea982684347aef0a8a1dc29c4ca P 04f18a0f77be4c61b338009521fda0d8: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260810 22:38:10.443639 7032 ts_tablet_manager.cc:1428] T fb4f0ea982684347aef0a8a1dc29c4ca P 84023a2618eb412598d0d0d07a70dbbc: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260810 22:38:10.443796 7033 ts_tablet_manager.cc:1428] T fb4f0ea982684347aef0a8a1dc29c4ca P 2efcb37f49184c2f9e10cb891c522ea2: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260810 22:38:10.444146 6728 heartbeater.cc:503] Master 127.16.233.126:36361 was elected leader, sending a full tablet report...
I20260810 22:38:10.444368 7014 heartbeater.cc:503] Master 127.16.233.126:36361 was elected leader, sending a full tablet report...
I20260810 22:38:10.597236 7040 raft_consensus.cc:482] T fb4f0ea982684347aef0a8a1dc29c4ca P 2efcb37f49184c2f9e10cb891c522ea2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:38:10.597466 7040 raft_consensus.cc:504] T fb4f0ea982684347aef0a8a1dc29c4ca P 2efcb37f49184c2f9e10cb891c522ea2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84023a2618eb412598d0d0d07a70dbbc" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 45187 } } peers { permanent_uuid: "04f18a0f77be4c61b338009521fda0d8" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 36535 } } peers { permanent_uuid: "2efcb37f49184c2f9e10cb891c522ea2" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 40817 } }
I20260810 22:38:10.598726 7040 leader_election.cc:288] T fb4f0ea982684347aef0a8a1dc29c4ca P 2efcb37f49184c2f9e10cb891c522ea2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84023a2618eb412598d0d0d07a70dbbc (127.16.233.67:45187), 04f18a0f77be4c61b338009521fda0d8 (127.16.233.66:36535)
I20260810 22:38:10.603559 7043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:10.599034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.66:36535 (local address 127.16.233.65:58613)
0810 22:38:10.599478 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:10.599490 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 22:38:10.599601 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:38:10.599979 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:38:10.599982 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:38:10.600001 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:38:10.600179 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:10.600186 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:10.601191 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:10.601195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:10.602317 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:10.602327 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:10.603025 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:10.603029 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:10.603150 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:10.603184 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 22:38:10.603264 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 22:38:10.603360 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":152,"threads_started":1}
W20260810 22:38:10.603526 6873 tablet.cc:2369] T fb4f0ea982684347aef0a8a1dc29c4ca P 04f18a0f77be4c61b338009521fda0d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:38:10.603544 7030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:10.599172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:58613 (local address 127.16.233.66:36535)
0810 22:38:10.599361 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:10.599368 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:10.599588 (+ 220us) server_negotiation.cc:407] Connection header received
0810 22:38:10.599697 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:10.599701 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:10.599756 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:10.599840 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:38:10.600348 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:10.601064 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:10.602465 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:10.602883 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:10.602982 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:10.603023 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:10.603297 (+ 274us) server_negotiation.cc:299] Negotiation successful
0810 22:38:10.603391 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 22:38:10.603964 7028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:10.599256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:41401 (local address 127.16.233.67:45187)
0810 22:38:10.599396 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:10.599400 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:10.599740 (+ 340us) server_negotiation.cc:407] Connection header received
0810 22:38:10.599988 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:10.599992 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:10.600078 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:10.600164 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:38:10.600665 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:10.601490 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:10.602554 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:10.603071 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:10.603201 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:10.603272 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:10.603726 (+ 454us) server_negotiation.cc:299] Negotiation successful
0810 22:38:10.603814 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 22:38:10.604327 7042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:10.599051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.67:45187 (local address 127.16.233.65:41401)
0810 22:38:10.599609 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:10.599625 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 22:38:10.599733 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:38:10.600291 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:38:10.600297 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:38:10.600322 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:38:10.600449 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:10.600456 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:10.601614 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:10.601617 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:10.602415 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:10.602428 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:10.603196 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:10.603198 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:10.603551 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:10.603582 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 22:38:10.604110 (+ 528us) client_negotiation.cc:240] Negotiation successful
0810 22:38:10.604197 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":128,"threads_started":1}
I20260810 22:38:10.604440 6824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb4f0ea982684347aef0a8a1dc29c4ca" candidate_uuid: "2efcb37f49184c2f9e10cb891c522ea2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04f18a0f77be4c61b338009521fda0d8" is_pre_election: true
I20260810 22:38:10.604801 6824 raft_consensus.cc:2454] T fb4f0ea982684347aef0a8a1dc29c4ca P 04f18a0f77be4c61b338009521fda0d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efcb37f49184c2f9e10cb891c522ea2 in term 0.
I20260810 22:38:10.605139 6966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb4f0ea982684347aef0a8a1dc29c4ca" candidate_uuid: "2efcb37f49184c2f9e10cb891c522ea2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84023a2618eb412598d0d0d07a70dbbc" is_pre_election: true
I20260810 22:38:10.605472 6966 raft_consensus.cc:2454] T fb4f0ea982684347aef0a8a1dc29c4ca P 84023a2618eb412598d0d0d07a70dbbc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efcb37f49184c2f9e10cb891c522ea2 in term 0.
I20260810 22:38:10.605448 6605 leader_election.cc:302] T fb4f0ea982684347aef0a8a1dc29c4ca P 2efcb37f49184c2f9e10cb891c522ea2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04f18a0f77be4c61b338009521fda0d8, 2efcb37f49184c2f9e10cb891c522ea2; no voters:
I20260810 22:38:10.605806 7040 raft_consensus.cc:2781] T fb4f0ea982684347aef0a8a1dc29c4ca P 2efcb37f49184c2f9e10cb891c522ea2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:38:10.605917 7040 raft_consensus.cc:482] T fb4f0ea982684347aef0a8a1dc29c4ca P 2efcb37f49184c2f9e10cb891c522ea2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:38:10.605976 7040 raft_consensus.cc:3037] T fb4f0ea982684347aef0a8a1dc29c4ca P 2efcb37f49184c2f9e10cb891c522ea2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:38:10.607112 7040 raft_consensus.cc:504] T fb4f0ea982684347aef0a8a1dc29c4ca P 2efcb37f49184c2f9e10cb891c522ea2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84023a2618eb412598d0d0d07a70dbbc" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 45187 } } peers { permanent_uuid: "04f18a0f77be4c61b338009521fda0d8" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 36535 } } peers { permanent_uuid: "2efcb37f49184c2f9e10cb891c522ea2" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 40817 } }
I20260810 22:38:10.607501 7040 leader_election.cc:288] T fb4f0ea982684347aef0a8a1dc29c4ca P 2efcb37f49184c2f9e10cb891c522ea2 [CANDIDATE]: Term 1 election: Requested vote from peers 84023a2618eb412598d0d0d07a70dbbc (127.16.233.67:45187), 04f18a0f77be4c61b338009521fda0d8 (127.16.233.66:36535)
I20260810 22:38:10.608009 6966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb4f0ea982684347aef0a8a1dc29c4ca" candidate_uuid: "2efcb37f49184c2f9e10cb891c522ea2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84023a2618eb412598d0d0d07a70dbbc"
I20260810 22:38:10.608101 6824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb4f0ea982684347aef0a8a1dc29c4ca" candidate_uuid: "2efcb37f49184c2f9e10cb891c522ea2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04f18a0f77be4c61b338009521fda0d8"
I20260810 22:38:10.608189 6966 raft_consensus.cc:3037] T fb4f0ea982684347aef0a8a1dc29c4ca P 84023a2618eb412598d0d0d07a70dbbc [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:38:10.608261 6824 raft_consensus.cc:3037] T fb4f0ea982684347aef0a8a1dc29c4ca P 04f18a0f77be4c61b338009521fda0d8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:38:10.609486 6824 raft_consensus.cc:2454] T fb4f0ea982684347aef0a8a1dc29c4ca P 04f18a0f77be4c61b338009521fda0d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efcb37f49184c2f9e10cb891c522ea2 in term 1.
I20260810 22:38:10.609648 6966 raft_consensus.cc:2454] T fb4f0ea982684347aef0a8a1dc29c4ca P 84023a2618eb412598d0d0d07a70dbbc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efcb37f49184c2f9e10cb891c522ea2 in term 1.
I20260810 22:38:10.609829 6605 leader_election.cc:302] T fb4f0ea982684347aef0a8a1dc29c4ca P 2efcb37f49184c2f9e10cb891c522ea2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04f18a0f77be4c61b338009521fda0d8, 2efcb37f49184c2f9e10cb891c522ea2; no voters:
I20260810 22:38:10.610013 7040 raft_consensus.cc:2781] T fb4f0ea982684347aef0a8a1dc29c4ca P 2efcb37f49184c2f9e10cb891c522ea2 [term 1 FOLLOWER]: Leader election won for term 1
W20260810 22:38:10.610066 7015 tablet.cc:2369] T fb4f0ea982684347aef0a8a1dc29c4ca P 84023a2618eb412598d0d0d07a70dbbc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:38:10.610293 7040 raft_consensus.cc:686] T fb4f0ea982684347aef0a8a1dc29c4ca P 2efcb37f49184c2f9e10cb891c522ea2 [term 1 LEADER]: Becoming Leader. State: Replica: 2efcb37f49184c2f9e10cb891c522ea2, State: Running, Role: LEADER
I20260810 22:38:10.610946 7040 consensus_queue.cc:237] T fb4f0ea982684347aef0a8a1dc29c4ca P 2efcb37f49184c2f9e10cb891c522ea2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84023a2618eb412598d0d0d07a70dbbc" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 45187 } } peers { permanent_uuid: "04f18a0f77be4c61b338009521fda0d8" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 36535 } } peers { permanent_uuid: "2efcb37f49184c2f9e10cb891c522ea2" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 40817 } }
W20260810 22:38:10.612300 6729 tablet.cc:2369] T fb4f0ea982684347aef0a8a1dc29c4ca P 2efcb37f49184c2f9e10cb891c522ea2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:38:10.614423 6521 catalog_manager.cc:5495] T fb4f0ea982684347aef0a8a1dc29c4ca P 2efcb37f49184c2f9e10cb891c522ea2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2efcb37f49184c2f9e10cb891c522ea2 (127.16.233.65). New cstate: current_term: 1 leader_uuid: "2efcb37f49184c2f9e10cb891c522ea2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84023a2618eb412598d0d0d07a70dbbc" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 45187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04f18a0f77be4c61b338009521fda0d8" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 36535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2efcb37f49184c2f9e10cb891c522ea2" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 40817 } health_report { overall_health: HEALTHY } } }
I20260810 22:38:10.665935 7029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:10.662249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40724 (local address 127.16.233.65:40817)
0810 22:38:10.662439 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:10.662448 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:10.662460 (+ 12us) server_negotiation.cc:407] Connection header received
0810 22:38:10.662541 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:10.662543 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:10.662601 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:10.662730 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:38:10.662984 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:10.663520 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:10.664397 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:10.664622 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:10.664740 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:10.665092 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:38:10.665527 (+ 435us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:38:10.665669 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:10.665711 (+ 42us) server_negotiation.cc:299] Negotiation successful
0810 22:38:10.665770 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 22:38:10.672040 6824 raft_consensus.cc:1261] T fb4f0ea982684347aef0a8a1dc29c4ca P 04f18a0f77be4c61b338009521fda0d8 [term 1 FOLLOWER]: Refusing update from remote peer 2efcb37f49184c2f9e10cb891c522ea2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:38:10.672035 6966 raft_consensus.cc:1261] T fb4f0ea982684347aef0a8a1dc29c4ca P 84023a2618eb412598d0d0d07a70dbbc [term 1 FOLLOWER]: Refusing update from remote peer 2efcb37f49184c2f9e10cb891c522ea2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:38:10.672842 7044 consensus_queue.cc:1035] T fb4f0ea982684347aef0a8a1dc29c4ca P 2efcb37f49184c2f9e10cb891c522ea2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84023a2618eb412598d0d0d07a70dbbc" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 45187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:38:10.673243 7040 consensus_queue.cc:1035] T fb4f0ea982684347aef0a8a1dc29c4ca P 2efcb37f49184c2f9e10cb891c522ea2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04f18a0f77be4c61b338009521fda0d8" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 36535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:38:10.682806 7048 mvcc.cc:205] Tried to move back new op lower bound from 7317100505784844288 to 7317100505545089024. Current Snapshot: MvccSnapshot[applied={T|T < 7317100505784844288}]
I20260810 22:38:10.691314 6520 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47434:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:38:10.691527 6520 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47434:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:38:10.694247 6520 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 13c2ab44d7094edc8b2c1d387f87d789: Sending DeleteTablet for 3 replicas of tablet fb4f0ea982684347aef0a8a1dc29c4ca
I20260810 22:38:10.695040 6942 tablet_service.cc:1514] Processing DeleteTablet for tablet fb4f0ea982684347aef0a8a1dc29c4ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:38:10 UTC) from {username='kudu', principal='kudu/127.16.233.126@KRBTEST.COM'} at 127.0.0.1:36950
I20260810 22:38:10.695238 6657 tablet_service.cc:1514] Processing DeleteTablet for tablet fb4f0ea982684347aef0a8a1dc29c4ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:38:10 UTC) from {username='kudu', principal='kudu/127.16.233.126@KRBTEST.COM'} at 127.0.0.1:40718
I20260810 22:38:10.695638 7058 tablet_replica.cc:330] T fb4f0ea982684347aef0a8a1dc29c4ca P 84023a2618eb412598d0d0d07a70dbbc: stopping tablet replica
I20260810 22:38:10.695683 6802 tablet_service.cc:1514] Processing DeleteTablet for tablet fb4f0ea982684347aef0a8a1dc29c4ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:38:10 UTC) from {username='kudu', principal='kudu/127.16.233.126@KRBTEST.COM'} at 127.0.0.1:48150
I20260810 22:38:10.696048 7058 raft_consensus.cc:2229] T fb4f0ea982684347aef0a8a1dc29c4ca P 84023a2618eb412598d0d0d07a70dbbc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:38:10.696180 7060 tablet_replica.cc:330] T fb4f0ea982684347aef0a8a1dc29c4ca P 04f18a0f77be4c61b338009521fda0d8: stopping tablet replica
I20260810 22:38:10.696444 7058 raft_consensus.cc:2258] T fb4f0ea982684347aef0a8a1dc29c4ca P 84023a2618eb412598d0d0d07a70dbbc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:38:10.696550 7060 raft_consensus.cc:2229] T fb4f0ea982684347aef0a8a1dc29c4ca P 04f18a0f77be4c61b338009521fda0d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:38:10.697000 7060 raft_consensus.cc:2258] T fb4f0ea982684347aef0a8a1dc29c4ca P 04f18a0f77be4c61b338009521fda0d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:38:10.697379 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 6589
I20260810 22:38:10.701324 7058 ts_tablet_manager.cc:1905] T fb4f0ea982684347aef0a8a1dc29c4ca P 84023a2618eb412598d0d0d07a70dbbc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:38:10.705090 7058 ts_tablet_manager.cc:1918] T fb4f0ea982684347aef0a8a1dc29c4ca P 84023a2618eb412598d0d0d07a70dbbc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:38:10.705191 7060 ts_tablet_manager.cc:1905] T fb4f0ea982684347aef0a8a1dc29c4ca P 04f18a0f77be4c61b338009521fda0d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:38:10.705252 7058 log.cc:1196] T fb4f0ea982684347aef0a8a1dc29c4ca P 84023a2618eb412598d0d0d07a70dbbc: Deleting WAL directory at /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/minicluster-data/ts-2/wal/wals/fb4f0ea982684347aef0a8a1dc29c4ca
I20260810 22:38:10.705611 7058 ts_tablet_manager.cc:1939] T fb4f0ea982684347aef0a8a1dc29c4ca P 84023a2618eb412598d0d0d07a70dbbc: Deleting consensus metadata
I20260810 22:38:10.707181 6507 catalog_manager.cc:4841] TS 84023a2618eb412598d0d0d07a70dbbc (127.16.233.67:45187): tablet fb4f0ea982684347aef0a8a1dc29c4ca (table test-table [id=6c1871b173014c27abd6a51d3dd7ed08]) successfully deleted
I20260810 22:38:10.707676 7060 ts_tablet_manager.cc:1918] T fb4f0ea982684347aef0a8a1dc29c4ca P 04f18a0f77be4c61b338009521fda0d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:38:10.707772 7060 log.cc:1196] T fb4f0ea982684347aef0a8a1dc29c4ca P 04f18a0f77be4c61b338009521fda0d8: Deleting WAL directory at /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786401215402979-17317-0/minicluster-data/ts-1/wal/wals/fb4f0ea982684347aef0a8a1dc29c4ca
I20260810 22:38:10.708070 7060 ts_tablet_manager.cc:1939] T fb4f0ea982684347aef0a8a1dc29c4ca P 04f18a0f77be4c61b338009521fda0d8: Deleting consensus metadata
I20260810 22:38:10.709380 6504 catalog_manager.cc:4841] TS 04f18a0f77be4c61b338009521fda0d8 (127.16.233.66:36535): tablet fb4f0ea982684347aef0a8a1dc29c4ca (table test-table [id=6c1871b173014c27abd6a51d3dd7ed08]) successfully deleted
W20260810 22:38:10.711040 6506 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.233.65:40817 (error 108)
I20260810 22:38:10.711292 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 6734
I20260810 22:38:10.713613 7026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:10.711797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.65:40817 (local address 127.0.0.1:40726)
0810 22:38:10.713415 (+ 1618us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:10.713502 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.65:40817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1504}
W20260810 22:38:10.713908 6506 catalog_manager.cc:4584] TS 2efcb37f49184c2f9e10cb891c522ea2 (127.16.233.65:40817): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fb4f0ea982684347aef0a8a1dc29c4ca: Network error: Client connection negotiation failed: client connection to 127.16.233.65:40817: connect: Connection refused (error 111)
I20260810 22:38:10.720121 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 6878
I20260810 22:38:10.728835 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 6486
2026-08-10T22:38:10Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2750 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-10T22:38:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:38:10Z Disabled control of system clock
I20260810 22:38:10.766880 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786401215402979-17317-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786401215402979-17317-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786401215402979-17317-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:46725
--webserver_interface=127.16.233.126
--webserver_port=0
--builtin_ntp_servers=127.16.233.84:35163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.126:46725
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 22:38:10.891314 7065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:38:10.891623 7065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:38:10.891660 7065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:38:10.895783 7065 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:38:10.895854 7065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:38:10.895869 7065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:38:10.895886 7065 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:38:10.895901 7065 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:38:10.900781 7065 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:35163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786401215402979-17317-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786401215402979-17317-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.126:46725
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:46725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786401215402979-17317-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:38:10.901968 7065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:38:10.903360 7065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:38:10.912537 7074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:10.912707 7072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:10.912595 7071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:38:10.912637 7065 server_base.cc:1034] running on GCE node
I20260810 22:38:10.913671 7065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:38:10.915096 7065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:38:10.916328 7065 hybrid_clock.cc:648] HybridClock initialized: now 1786401490916321 us; error 51 us; skew 500 ppm
I20260810 22:38:10.918596 7065 webserver.cc:460] Webserver started at http://127.16.233.126:34789/ using document root <none> and password file <none>
I20260810 22:38:10.919248 7065 fs_manager.cc:359] Metadata directory not provided
I20260810 22:38:10.919378 7065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:38:10.919687 7065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:38:10.921921 7065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786401215402979-17317-0/minicluster-data/master-0/data/instance:
uuid: "76ca050f630a47a1b919fc55e1f25183"
format_stamp: "Formatted at 2026-08-10 22:38:10 on dist-test-slave-77v9"
server_key: "b9e266a864977abc2b2901b219ba4f54"
server_key_iv: "309bd559e35e35c4d0493ba5cfd7206a"
server_key_version: "encryptionkey@0"
I20260810 22:38:10.922484 7065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786401215402979-17317-0/minicluster-data/master-0/wal/instance:
uuid: "76ca050f630a47a1b919fc55e1f25183"
format_stamp: "Formatted at 2026-08-10 22:38:10 on dist-test-slave-77v9"
server_key: "b9e266a864977abc2b2901b219ba4f54"
server_key_iv: "309bd559e35e35c4d0493ba5cfd7206a"
server_key_version: "encryptionkey@0"
I20260810 22:38:10.926388 7065 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 22:38:10.929065 7081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:10.930418 7065 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:38:10.930533 7065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786401215402979-17317-0/minicluster-data/master-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786401215402979-17317-0/minicluster-data/master-0/wal
uuid: "76ca050f630a47a1b919fc55e1f25183"
format_stamp: "Formatted at 2026-08-10 22:38:10 on dist-test-slave-77v9"
server_key: "b9e266a864977abc2b2901b219ba4f54"
server_key_iv: "309bd559e35e35c4d0493ba5cfd7206a"
server_key_version: "encryptionkey@0"
I20260810 22:38:10.930632 7065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786401215402979-17317-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786401215402979-17317-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786401215402979-17317-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:38:10.946573 7065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:38:10.947319 7065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:38:10.947499 7065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:38:10.955259 7138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.126:46725 every 8 connection(s)
I20260810 22:38:10.955278 7065 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.126:46725
I20260810 22:38:10.956513 7065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
I20260810 22:38:10.960040 7139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:38:10.964783 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 7065
I20260810 22:38:10.964967 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786401215402979-17317-0/minicluster-data/master-0/wal/instance
I20260810 22:38:10.965299 17317 external_mini_cluster.cc:1428] Setting key 93c84c824ebd509601032b983390657e
I20260810 22:38:10.966372 7139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76ca050f630a47a1b919fc55e1f25183: Bootstrap starting.
I20260810 22:38:10.968883 7139 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76ca050f630a47a1b919fc55e1f25183: Neither blocks nor log segments found. Creating new log.
I20260810 22:38:10.969835 7139 log.cc:824] T 00000000000000000000000000000000 P 76ca050f630a47a1b919fc55e1f25183: Log is configured to *not* fsync() on all Append() calls
I20260810 22:38:10.972110 7139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76ca050f630a47a1b919fc55e1f25183: No bootstrap required, opened a new log
I20260810 22:38:10.975402 7142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:10.966933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51258 (local address 127.16.233.126:46725)
0810 22:38:10.967630 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:10.967647 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:10.967685 (+ 38us) server_negotiation.cc:407] Connection header received
0810 22:38:10.968356 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:10.968373 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:10.968801 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:10.969218 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:10.969583 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:10.970610 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:10.971818 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:10.972377 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:10.972500 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:10.973459 (+ 959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:10.973464 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:10.973477 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:38:10.973951 (+ 474us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:10.974011 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:38:10.974032 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:10.974116 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:10.974315 (+ 199us) server_negotiation.cc:299] Negotiation successful
0810 22:38:10.974648 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":170,"threads_started":1}
I20260810 22:38:10.976289 7139 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76ca050f630a47a1b919fc55e1f25183 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76ca050f630a47a1b919fc55e1f25183" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 46725 } }
I20260810 22:38:10.976575 7139 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76ca050f630a47a1b919fc55e1f25183 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:38:10.976646 7139 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76ca050f630a47a1b919fc55e1f25183 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76ca050f630a47a1b919fc55e1f25183, State: Initialized, Role: FOLLOWER
I20260810 22:38:10.977149 7139 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76ca050f630a47a1b919fc55e1f25183 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76ca050f630a47a1b919fc55e1f25183" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 46725 } }
I20260810 22:38:10.977303 7139 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76ca050f630a47a1b919fc55e1f25183 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:38:10.977367 7139 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76ca050f630a47a1b919fc55e1f25183 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:38:10.977469 7139 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76ca050f630a47a1b919fc55e1f25183 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:38:10.978438 7139 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76ca050f630a47a1b919fc55e1f25183 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76ca050f630a47a1b919fc55e1f25183" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 46725 } }
I20260810 22:38:10.978871 7139 leader_election.cc:302] T 00000000000000000000000000000000 P 76ca050f630a47a1b919fc55e1f25183 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76ca050f630a47a1b919fc55e1f25183; no voters:
I20260810 22:38:10.979184 7139 leader_election.cc:288] T 00000000000000000000000000000000 P 76ca050f630a47a1b919fc55e1f25183 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:38:10.979331 7144 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76ca050f630a47a1b919fc55e1f25183 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:38:10.979605 7144 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76ca050f630a47a1b919fc55e1f25183 [term 1 LEADER]: Becoming Leader. State: Replica: 76ca050f630a47a1b919fc55e1f25183, State: Running, Role: LEADER
I20260810 22:38:10.979945 7144 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76ca050f630a47a1b919fc55e1f25183 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76ca050f630a47a1b919fc55e1f25183" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 46725 } }
I20260810 22:38:10.980437 7139 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76ca050f630a47a1b919fc55e1f25183 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:38:10.981786 7146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76ca050f630a47a1b919fc55e1f25183 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76ca050f630a47a1b919fc55e1f25183" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76ca050f630a47a1b919fc55e1f25183" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 46725 } } }
I20260810 22:38:10.981797 7147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76ca050f630a47a1b919fc55e1f25183 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76ca050f630a47a1b919fc55e1f25183. Latest consensus state: current_term: 1 leader_uuid: "76ca050f630a47a1b919fc55e1f25183" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76ca050f630a47a1b919fc55e1f25183" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 46725 } } }
I20260810 22:38:10.982012 7147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76ca050f630a47a1b919fc55e1f25183 [sys.catalog]: This master's current role is: LEADER
I20260810 22:38:10.981966 7146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76ca050f630a47a1b919fc55e1f25183 [sys.catalog]: This master's current role is: LEADER
W20260810 22:38:10.985399 7162 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 76ca050f630a47a1b919fc55e1f25183: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 22:38:10.985468 7162 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 22:38:10.985544 7161 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:38:10.986222 7161 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:38:10.992394 7161 catalog_manager.cc:1347] Generated new cluster ID: 5c31bd4fac8248db92e0a9a1135b651b
I20260810 22:38:10.992486 7161 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:38:11.000603 7161 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:38:11.001895 7161 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:38:11.009675 7161 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76ca050f630a47a1b919fc55e1f25183: Generated new TSK 0
I20260810 22:38:11.010612 7161 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:38:11.014307 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786401215402979-17317-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786401215402979-17317-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786401215402979-17317-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.65:0
--local_ip_for_outbound_sockets=127.16.233.65
--webserver_interface=127.16.233.65
--webserver_port=0
--tserver_master_addrs=127.16.233.126:46725
--builtin_ntp_servers=127.16.233.84:35163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 22:38:11.138908 7166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:38:11.139209 7166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:38:11.139252 7166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:38:11.143496 7166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:38:11.143568 7166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:38:11.143626 7166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.65
I20260810 22:38:11.148584 7166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:35163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786401215402979-17317-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786401215402979-17317-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.233.65
--webserver_port=0
--tserver_master_addrs=127.16.233.126:46725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.65
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786401215402979-17317-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:38:11.149834 7166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:38:11.151222 7166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:38:11.161615 7175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:11.161847 7172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:38:11.161813 7166 server_base.cc:1034] running on GCE node
W20260810 22:38:11.161620 7171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:38:11.162482 7166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:38:11.163209 7166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:38:11.164438 7166 hybrid_clock.cc:648] HybridClock initialized: now 1786401491164421 us; error 35 us; skew 500 ppm
I20260810 22:38:11.166484 7166 webserver.cc:460] Webserver started at http://127.16.233.65:38601/ using document root <none> and password file <none>
I20260810 22:38:11.167095 7166 fs_manager.cc:359] Metadata directory not provided
I20260810 22:38:11.167196 7166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:38:11.167438 7166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:38:11.169656 7166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786401215402979-17317-0/minicluster-data/ts-0/data/instance:
uuid: "c4726169d9b44c9ead2cc7fcff5d383c"
format_stamp: "Formatted at 2026-08-10 22:38:11 on dist-test-slave-77v9"
server_key: "9014156f53d0251f10372af9099195ad"
server_key_iv: "7b3e4bf07b689eda6678a1b3fa1c439d"
server_key_version: "encryptionkey@0"
I20260810 22:38:11.170302 7166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786401215402979-17317-0/minicluster-data/ts-0/wal/instance:
uuid: "c4726169d9b44c9ead2cc7fcff5d383c"
format_stamp: "Formatted at 2026-08-10 22:38:11 on dist-test-slave-77v9"
server_key: "9014156f53d0251f10372af9099195ad"
server_key_iv: "7b3e4bf07b689eda6678a1b3fa1c439d"
server_key_version: "encryptionkey@0"
I20260810 22:38:11.174317 7166 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 22:38:11.177134 7181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:11.178544 7166 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:38:11.178694 7166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786401215402979-17317-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786401215402979-17317-0/minicluster-data/ts-0/wal
uuid: "c4726169d9b44c9ead2cc7fcff5d383c"
format_stamp: "Formatted at 2026-08-10 22:38:11 on dist-test-slave-77v9"
server_key: "9014156f53d0251f10372af9099195ad"
server_key_iv: "7b3e4bf07b689eda6678a1b3fa1c439d"
server_key_version: "encryptionkey@0"
I20260810 22:38:11.178821 7166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786401215402979-17317-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786401215402979-17317-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786401215402979-17317-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:38:11.197511 7166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:38:11.198328 7166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:38:11.198575 7166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:38:11.199317 7166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:38:11.200439 7166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:38:11.200515 7166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:11.200584 7166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:38:11.200636 7166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:11.210414 7166 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.65:44759
I20260810 22:38:11.210453 7301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.65:44759 every 8 connection(s)
I20260810 22:38:11.211427 7166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786401215402979-17317-0/minicluster-data/ts-0/data/info.pb
I20260810 22:38:11.213153 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 7166
I20260810 22:38:11.213328 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786401215402979-17317-0/minicluster-data/ts-0/wal/instance
I20260810 22:38:11.213745 17317 external_mini_cluster.cc:1428] Setting key ba3e3f4579fa0f353a1d00d323bbbf87
I20260810 22:38:11.215929 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786401215402979-17317-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786401215402979-17317-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786401215402979-17317-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.66:0
--local_ip_for_outbound_sockets=127.16.233.66
--webserver_interface=127.16.233.66
--webserver_port=0
--tserver_master_addrs=127.16.233.126:46725
--builtin_ntp_servers=127.16.233.84:35163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 22:38:11.221732 7142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:11.213670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.65:44855 (local address 127.16.233.126:46725)
0810 22:38:11.213846 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:11.213851 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:11.215088 (+ 1237us) server_negotiation.cc:407] Connection header received
0810 22:38:11.216685 (+ 1597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:11.216688 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:11.216746 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:11.216850 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:11.217822 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:11.218376 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:11.219552 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:11.219784 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:11.219908 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:11.221203 (+ 1295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:11.221207 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:11.221209 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:38:11.221230 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:11.221249 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:38:11.221258 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:11.221340 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:11.221509 (+ 169us) server_negotiation.cc:299] Negotiation successful
0810 22:38:11.221577 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 22:38:11.222656 7304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:11.214133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:46725 (local address 127.16.233.65:44855)
0810 22:38:11.214899 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:11.214957 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 22:38:11.216447 (+ 1490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:38:11.217033 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:38:11.217039 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:38:11.217127 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:38:11.217666 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:11.217679 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:11.218514 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:11.218518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:11.219411 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:11.219420 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:11.219968 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:11.219973 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:11.220191 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:11.220921 (+ 730us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 22:38:11.220923 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:38:11.220966 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 22:38:11.221035 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:38:11.221341 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:38:11.221354 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:38:11.221362 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 22:38:11.221508 (+ 146us) client_negotiation.cc:240] Negotiation successful
0810 22:38:11.221815 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":154,"threads_started":1}
I20260810 22:38:11.224033 7302 heartbeater.cc:347] Connected to a master server at 127.16.233.126:46725
I20260810 22:38:11.224298 7302 heartbeater.cc:464] Registering TS with master...
I20260810 22:38:11.224924 7302 heartbeater.cc:511] Master 127.16.233.126:46725 requested a full tablet report, sending...
I20260810 22:38:11.226790 7100 ts_manager.cc:194] Registered new tserver with Master: c4726169d9b44c9ead2cc7fcff5d383c (127.16.233.65:44759)
I20260810 22:38:11.228837 7100 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.233.65:44855
W20260810 22:38:11.339385 7305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:38:11.339695 7305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:38:11.339745 7305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:38:11.343855 7305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:38:11.343930 7305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:38:11.343991 7305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.66
I20260810 22:38:11.348994 7305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:35163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786401215402979-17317-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786401215402979-17317-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.233.66
--webserver_port=0
--tserver_master_addrs=127.16.233.126:46725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.66
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786401215402979-17317-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:38:11.350195 7305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:38:11.351591 7305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:38:11.362089 7310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:11.362089 7311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:11.362102 7313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:38:11.362416 7305 server_base.cc:1034] running on GCE node
I20260810 22:38:11.362965 7305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:38:11.363688 7305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:38:11.364890 7305 hybrid_clock.cc:648] HybridClock initialized: now 1786401491364875 us; error 34 us; skew 500 ppm
I20260810 22:38:11.366968 7305 webserver.cc:460] Webserver started at http://127.16.233.66:38689/ using document root <none> and password file <none>
I20260810 22:38:11.367600 7305 fs_manager.cc:359] Metadata directory not provided
I20260810 22:38:11.367702 7305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:38:11.367967 7305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:38:11.370203 7305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786401215402979-17317-0/minicluster-data/ts-1/data/instance:
uuid: "2cb59bb34c2c442da48fb5a3a60ac8b2"
format_stamp: "Formatted at 2026-08-10 22:38:11 on dist-test-slave-77v9"
server_key: "025e7d6990b962e6e45e845b706351cb"
server_key_iv: "581bfd7f6d1e4beb2658741d9763309c"
server_key_version: "encryptionkey@0"
I20260810 22:38:11.370812 7305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786401215402979-17317-0/minicluster-data/ts-1/wal/instance:
uuid: "2cb59bb34c2c442da48fb5a3a60ac8b2"
format_stamp: "Formatted at 2026-08-10 22:38:11 on dist-test-slave-77v9"
server_key: "025e7d6990b962e6e45e845b706351cb"
server_key_iv: "581bfd7f6d1e4beb2658741d9763309c"
server_key_version: "encryptionkey@0"
I20260810 22:38:11.374738 7305 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260810 22:38:11.377395 7320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:11.378772 7305 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:38:11.378931 7305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786401215402979-17317-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786401215402979-17317-0/minicluster-data/ts-1/wal
uuid: "2cb59bb34c2c442da48fb5a3a60ac8b2"
format_stamp: "Formatted at 2026-08-10 22:38:11 on dist-test-slave-77v9"
server_key: "025e7d6990b962e6e45e845b706351cb"
server_key_iv: "581bfd7f6d1e4beb2658741d9763309c"
server_key_version: "encryptionkey@0"
I20260810 22:38:11.379058 7305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786401215402979-17317-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786401215402979-17317-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786401215402979-17317-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:38:11.396468 7305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:38:11.397389 7305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:38:11.397640 7305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:38:11.398371 7305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:38:11.399498 7305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:38:11.399580 7305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:11.399663 7305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:38:11.399711 7305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:11.410688 7305 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.66:40307
I20260810 22:38:11.410697 7440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.66:40307 every 8 connection(s)
I20260810 22:38:11.411820 7305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786401215402979-17317-0/minicluster-data/ts-1/data/info.pb
I20260810 22:38:11.414605 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 7305
I20260810 22:38:11.414717 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786401215402979-17317-0/minicluster-data/ts-1/wal/instance
I20260810 22:38:11.415050 17317 external_mini_cluster.cc:1428] Setting key 28745743ba9348ccce74ae715a497be1
I20260810 22:38:11.417346 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786401215402979-17317-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786401215402979-17317-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786401215402979-17317-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.233.67:0
--local_ip_for_outbound_sockets=127.16.233.67
--webserver_interface=127.16.233.67
--webserver_port=0
--tserver_master_addrs=127.16.233.126:46725
--builtin_ntp_servers=127.16.233.84:35163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 22:38:11.422467 7142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:11.414080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:48745 (local address 127.16.233.126:46725)
0810 22:38:11.414265 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:11.414269 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:11.415476 (+ 1207us) server_negotiation.cc:407] Connection header received
0810 22:38:11.417204 (+ 1728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:11.417208 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:11.417267 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:11.417378 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:11.418331 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:11.418872 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:11.420115 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:11.420369 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:11.420472 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:11.421906 (+ 1434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:11.421909 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:11.421911 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:38:11.421924 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:11.421943 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:38:11.421951 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:11.422010 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:11.422213 (+ 203us) server_negotiation.cc:299] Negotiation successful
0810 22:38:11.422267 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 22:38:11.423840 7443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:11.414480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:46725 (local address 127.16.233.66:48745)
0810 22:38:11.415266 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:11.415321 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 22:38:11.416926 (+ 1605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:38:11.417567 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:38:11.417575 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:38:11.417661 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:38:11.418186 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:11.418198 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:11.419017 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:11.419021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:11.419964 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:11.419974 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:11.420595 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:11.420602 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:11.420881 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:11.421569 (+ 688us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 22:38:11.421574 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:38:11.421640 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 22:38:11.421729 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:38:11.422044 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:38:11.422057 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:38:11.422067 (+ 10us) client_negotiation.cc:769] Sending connection context
0810 22:38:11.422210 (+ 143us) client_negotiation.cc:240] Negotiation successful
0810 22:38:11.422554 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":161,"threads_started":1}
I20260810 22:38:11.425158 7441 heartbeater.cc:347] Connected to a master server at 127.16.233.126:46725
I20260810 22:38:11.425436 7441 heartbeater.cc:464] Registering TS with master...
I20260810 22:38:11.426185 7441 heartbeater.cc:511] Master 127.16.233.126:46725 requested a full tablet report, sending...
I20260810 22:38:11.427594 7100 ts_manager.cc:194] Registered new tserver with Master: 2cb59bb34c2c442da48fb5a3a60ac8b2 (127.16.233.66:40307)
I20260810 22:38:11.428532 7100 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.233.66:48745
W20260810 22:38:11.544251 7444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:38:11.544584 7444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:38:11.544629 7444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:38:11.549396 7444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:38:11.549479 7444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:38:11.549544 7444 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.233.67
I20260810 22:38:11.555094 7444 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:35163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786401215402979-17317-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786401215402979-17317-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.233.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.233.67
--webserver_port=0
--tserver_master_addrs=127.16.233.126:46725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.233.67
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786401215402979-17317-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:38:11.556308 7444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:38:11.557812 7444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:38:11.568167 7450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:11.568244 7449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:38:11.568289 7444 server_base.cc:1034] running on GCE node
W20260810 22:38:11.568496 7452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:38:11.568962 7444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:38:11.569600 7444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:38:11.570787 7444 hybrid_clock.cc:648] HybridClock initialized: now 1786401491570748 us; error 54 us; skew 500 ppm
I20260810 22:38:11.572961 7444 webserver.cc:460] Webserver started at http://127.16.233.67:38407/ using document root <none> and password file <none>
I20260810 22:38:11.573587 7444 fs_manager.cc:359] Metadata directory not provided
I20260810 22:38:11.573655 7444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:38:11.573880 7444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:38:11.576426 7444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786401215402979-17317-0/minicluster-data/ts-2/data/instance:
uuid: "e5482093ec7543069fc6c562145077de"
format_stamp: "Formatted at 2026-08-10 22:38:11 on dist-test-slave-77v9"
server_key: "34cb93a82178517df095d72245fb1947"
server_key_iv: "9d67e6e22db81446077830b8eec84f7f"
server_key_version: "encryptionkey@0"
I20260810 22:38:11.577195 7444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786401215402979-17317-0/minicluster-data/ts-2/wal/instance:
uuid: "e5482093ec7543069fc6c562145077de"
format_stamp: "Formatted at 2026-08-10 22:38:11 on dist-test-slave-77v9"
server_key: "34cb93a82178517df095d72245fb1947"
server_key_iv: "9d67e6e22db81446077830b8eec84f7f"
server_key_version: "encryptionkey@0"
I20260810 22:38:11.581535 7444 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260810 22:38:11.584690 7458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:11.586431 7444 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 22:38:11.586633 7444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786401215402979-17317-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786401215402979-17317-0/minicluster-data/ts-2/wal
uuid: "e5482093ec7543069fc6c562145077de"
format_stamp: "Formatted at 2026-08-10 22:38:11 on dist-test-slave-77v9"
server_key: "34cb93a82178517df095d72245fb1947"
server_key_iv: "9d67e6e22db81446077830b8eec84f7f"
server_key_version: "encryptionkey@0"
I20260810 22:38:11.586777 7444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786401215402979-17317-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786401215402979-17317-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786401215402979-17317-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:38:11.604780 7444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:38:11.605611 7444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:38:11.605860 7444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:38:11.606564 7444 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:38:11.607612 7444 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:38:11.607688 7444 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:11.607757 7444 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:38:11.607807 7444 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:11.617385 7444 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.67:35179
I20260810 22:38:11.617439 7578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.67:35179 every 8 connection(s)
I20260810 22:38:11.618359 7444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786401215402979-17317-0/minicluster-data/ts-2/data/info.pb
I20260810 22:38:11.626423 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 7444
I20260810 22:38:11.626573 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786401215402979-17317-0/minicluster-data/ts-2/wal/instance
I20260810 22:38:11.626909 17317 external_mini_cluster.cc:1428] Setting key 1ee1b9820b527b57dabffd086fd1336d
I20260810 22:38:11.628713 7142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:11.620418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.67:54675 (local address 127.16.233.126:46725)
0810 22:38:11.620602 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:11.620606 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:11.621942 (+ 1336us) server_negotiation.cc:407] Connection header received
0810 22:38:11.623333 (+ 1391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:11.623336 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:11.623388 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:11.623472 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:11.624471 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:11.625097 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:11.626303 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:11.626672 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:11.626798 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:11.628120 (+ 1322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:11.628123 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:11.628125 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:38:11.628137 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:11.628177 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:38:11.628188 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:11.628274 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:11.628494 (+ 220us) server_negotiation.cc:299] Negotiation successful
0810 22:38:11.628557 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260810 22:38:11.629772 7582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:11.620880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:46725 (local address 127.16.233.67:54675)
0810 22:38:11.621693 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:11.621747 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 22:38:11.623128 (+ 1381us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:38:11.623655 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:38:11.623661 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:38:11.623746 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:38:11.624325 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:11.624338 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:11.625229 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:11.625231 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:11.626157 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:11.626167 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:11.626905 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:11.626908 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:11.627108 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:11.627828 (+ 720us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 22:38:11.627830 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:38:11.627872 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 22:38:11.627970 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:38:11.628303 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:38:11.628315 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:38:11.628322 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 22:38:11.628463 (+ 141us) client_negotiation.cc:240] Negotiation successful
0810 22:38:11.628710 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":173,"threads_started":1}
I20260810 22:38:11.630903 7579 heartbeater.cc:347] Connected to a master server at 127.16.233.126:46725
I20260810 22:38:11.631109 7579 heartbeater.cc:464] Registering TS with master...
I20260810 22:38:11.631601 7579 heartbeater.cc:511] Master 127.16.233.126:46725 requested a full tablet report, sending...
I20260810 22:38:11.632686 7100 ts_manager.cc:194] Registered new tserver with Master: e5482093ec7543069fc6c562145077de (127.16.233.67:35179)
I20260810 22:38:11.633404 7100 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.233.67:54675
I20260810 22:38:11.641444 17317 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 22:38:11.648036 7142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:11.643632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51266 (local address 127.16.233.126:46725)
0810 22:38:11.643792 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:11.643797 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:11.643870 (+ 73us) server_negotiation.cc:407] Connection header received
0810 22:38:11.644042 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:11.644045 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:11.644096 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:11.644180 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:11.644478 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:11.645213 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:11.646101 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:11.646371 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:11.646490 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:11.647552 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:11.647555 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:11.647558 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:38:11.647572 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:11.647593 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:38:11.647602 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:11.647670 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:11.647830 (+ 160us) server_negotiation.cc:299] Negotiation successful
0810 22:38:11.647896 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 22:38:11.651683 7100 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51266:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 22:38:11.654779 7100 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:38:11.671887 7595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:11.665935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58632 (local address 127.16.233.66:40307)
0810 22:38:11.666493 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:11.666510 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:11.666539 (+ 29us) server_negotiation.cc:407] Connection header received
0810 22:38:11.666629 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:11.666634 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:11.666858 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:11.667043 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:38:11.667340 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:11.668683 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:11.670124 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:11.670967 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:11.671275 (+ 308us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:11.671487 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:11.671552 (+ 65us) server_negotiation.cc:299] Negotiation successful
0810 22:38:11.671658 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":150,"threads_started":1}
I20260810 22:38:11.671891 7594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:11.665909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.65:44759 (local address 127.0.0.1:37466)
0810 22:38:11.666314 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:11.666328 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:38:11.666412 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:38:11.667237 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:38:11.667241 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:38:11.667262 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:38:11.667366 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:11.667374 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:11.668890 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:11.668894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:11.670190 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:11.670201 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:11.671113 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:11.671118 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:11.671477 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:11.671510 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 22:38:11.671600 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 22:38:11.671699 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"mutex_wait_us":182,"thread_start_us":94,"threads_started":1}
I20260810 22:38:11.672152 7591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:11.664704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.67:35179 (local address 127.0.0.1:51422)
0810 22:38:11.665300 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:11.665356 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 22:38:11.665501 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:38:11.666181 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:38:11.666189 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:38:11.666241 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:38:11.666439 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:11.666466 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:11.668187 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:11.668191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:11.669861 (+ 1670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:11.669875 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:11.671132 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:11.671136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:11.671809 (+ 673us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:11.671832 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 22:38:11.671906 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 22:38:11.671987 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"mutex_wait_us":80,"thread_start_us":139,"threads_started":1}
I20260810 22:38:11.672655 7593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:11.665542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.66:40307 (local address 127.0.0.1:58632)
0810 22:38:11.665988 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:11.666005 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 22:38:11.666099 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:38:11.667032 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:38:11.667035 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:38:11.667053 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:38:11.667177 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:11.667187 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:11.668894 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:11.668901 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:11.669923 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:11.669934 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:11.671132 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:11.671136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:11.671355 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:11.671407 (+ 52us) client_negotiation.cc:769] Sending connection context
0810 22:38:11.671526 (+ 119us) client_negotiation.cc:240] Negotiation successful
0810 22:38:11.671663 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"mutex_wait_us":148,"thread_start_us":112,"threads_started":1}
I20260810 22:38:11.672680 7592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:11.664986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51422 (local address 127.16.233.67:35179)
0810 22:38:11.665536 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:11.665553 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:11.665603 (+ 50us) server_negotiation.cc:407] Connection header received
0810 22:38:11.665698 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:11.665704 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:11.665936 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:11.666663 (+ 727us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:38:11.666836 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:11.668033 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:11.670037 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:11.670964 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:11.672135 (+ 1171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:11.672357 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:11.672410 (+ 53us) server_negotiation.cc:299] Negotiation successful
0810 22:38:11.672500 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":157,"threads_started":1}
I20260810 22:38:11.673237 7596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:11.666393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37466 (local address 127.16.233.65:44759)
0810 22:38:11.666796 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:11.666806 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:11.666827 (+ 21us) server_negotiation.cc:407] Connection header received
0810 22:38:11.666892 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:11.666896 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:11.667089 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:11.667260 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:38:11.667513 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:11.668736 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:11.670344 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:11.670957 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:11.671113 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:11.671371 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:11.672962 (+ 1591us) server_negotiation.cc:299] Negotiation successful
0810 22:38:11.673075 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":183,"threads_started":1}
I20260810 22:38:11.675828 7371 tablet_service.cc:1467] Processing CreateTablet for tablet a46f5c843378412685484ba45615138a (DEFAULT_TABLE table=test-table [id=a0186666ebd4410fa6828259272f33dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:38:11.676054 7230 tablet_service.cc:1467] Processing CreateTablet for tablet a46f5c843378412685484ba45615138a (DEFAULT_TABLE table=test-table [id=a0186666ebd4410fa6828259272f33dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:38:11.676054 7508 tablet_service.cc:1467] Processing CreateTablet for tablet a46f5c843378412685484ba45615138a (DEFAULT_TABLE table=test-table [id=a0186666ebd4410fa6828259272f33dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:38:11.677071 7371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a46f5c843378412685484ba45615138a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:38:11.677222 7508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a46f5c843378412685484ba45615138a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:38:11.677227 7230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a46f5c843378412685484ba45615138a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:38:11.685142 7597 tablet_bootstrap.cc:492] T a46f5c843378412685484ba45615138a P 2cb59bb34c2c442da48fb5a3a60ac8b2: Bootstrap starting.
I20260810 22:38:11.686519 7598 tablet_bootstrap.cc:492] T a46f5c843378412685484ba45615138a P c4726169d9b44c9ead2cc7fcff5d383c: Bootstrap starting.
I20260810 22:38:11.686885 7599 tablet_bootstrap.cc:492] T a46f5c843378412685484ba45615138a P e5482093ec7543069fc6c562145077de: Bootstrap starting.
I20260810 22:38:11.687820 7597 tablet_bootstrap.cc:654] T a46f5c843378412685484ba45615138a P 2cb59bb34c2c442da48fb5a3a60ac8b2: Neither blocks nor log segments found. Creating new log.
I20260810 22:38:11.688884 7597 log.cc:824] T a46f5c843378412685484ba45615138a P 2cb59bb34c2c442da48fb5a3a60ac8b2: Log is configured to *not* fsync() on all Append() calls
I20260810 22:38:11.689149 7598 tablet_bootstrap.cc:654] T a46f5c843378412685484ba45615138a P c4726169d9b44c9ead2cc7fcff5d383c: Neither blocks nor log segments found. Creating new log.
I20260810 22:38:11.689198 7599 tablet_bootstrap.cc:654] T a46f5c843378412685484ba45615138a P e5482093ec7543069fc6c562145077de: Neither blocks nor log segments found. Creating new log.
I20260810 22:38:11.690089 7599 log.cc:824] T a46f5c843378412685484ba45615138a P e5482093ec7543069fc6c562145077de: Log is configured to *not* fsync() on all Append() calls
I20260810 22:38:11.690089 7598 log.cc:824] T a46f5c843378412685484ba45615138a P c4726169d9b44c9ead2cc7fcff5d383c: Log is configured to *not* fsync() on all Append() calls
I20260810 22:38:11.692513 7597 tablet_bootstrap.cc:492] T a46f5c843378412685484ba45615138a P 2cb59bb34c2c442da48fb5a3a60ac8b2: No bootstrap required, opened a new log
I20260810 22:38:11.692741 7597 ts_tablet_manager.cc:1397] T a46f5c843378412685484ba45615138a P 2cb59bb34c2c442da48fb5a3a60ac8b2: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260810 22:38:11.693305 7599 tablet_bootstrap.cc:492] T a46f5c843378412685484ba45615138a P e5482093ec7543069fc6c562145077de: No bootstrap required, opened a new log
I20260810 22:38:11.693457 7598 tablet_bootstrap.cc:492] T a46f5c843378412685484ba45615138a P c4726169d9b44c9ead2cc7fcff5d383c: No bootstrap required, opened a new log
I20260810 22:38:11.693518 7599 ts_tablet_manager.cc:1397] T a46f5c843378412685484ba45615138a P e5482093ec7543069fc6c562145077de: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 22:38:11.693668 7598 ts_tablet_manager.cc:1397] T a46f5c843378412685484ba45615138a P c4726169d9b44c9ead2cc7fcff5d383c: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 22:38:11.697280 7598 raft_consensus.cc:348] T a46f5c843378412685484ba45615138a P c4726169d9b44c9ead2cc7fcff5d383c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5482093ec7543069fc6c562145077de" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 35179 } } peers { permanent_uuid: "2cb59bb34c2c442da48fb5a3a60ac8b2" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 40307 } } peers { permanent_uuid: "c4726169d9b44c9ead2cc7fcff5d383c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 44759 } }
I20260810 22:38:11.697561 7598 raft_consensus.cc:374] T a46f5c843378412685484ba45615138a P c4726169d9b44c9ead2cc7fcff5d383c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:38:11.697619 7598 raft_consensus.cc:729] T a46f5c843378412685484ba45615138a P c4726169d9b44c9ead2cc7fcff5d383c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4726169d9b44c9ead2cc7fcff5d383c, State: Initialized, Role: FOLLOWER
I20260810 22:38:11.697814 7597 raft_consensus.cc:348] T a46f5c843378412685484ba45615138a P 2cb59bb34c2c442da48fb5a3a60ac8b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5482093ec7543069fc6c562145077de" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 35179 } } peers { permanent_uuid: "2cb59bb34c2c442da48fb5a3a60ac8b2" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 40307 } } peers { permanent_uuid: "c4726169d9b44c9ead2cc7fcff5d383c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 44759 } }
I20260810 22:38:11.698117 7597 raft_consensus.cc:374] T a46f5c843378412685484ba45615138a P 2cb59bb34c2c442da48fb5a3a60ac8b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:38:11.698189 7597 raft_consensus.cc:729] T a46f5c843378412685484ba45615138a P 2cb59bb34c2c442da48fb5a3a60ac8b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cb59bb34c2c442da48fb5a3a60ac8b2, State: Initialized, Role: FOLLOWER
I20260810 22:38:11.698115 7598 consensus_queue.cc:260] T a46f5c843378412685484ba45615138a P c4726169d9b44c9ead2cc7fcff5d383c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5482093ec7543069fc6c562145077de" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 35179 } } peers { permanent_uuid: "2cb59bb34c2c442da48fb5a3a60ac8b2" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 40307 } } peers { permanent_uuid: "c4726169d9b44c9ead2cc7fcff5d383c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 44759 } }
I20260810 22:38:11.698112 7599 raft_consensus.cc:348] T a46f5c843378412685484ba45615138a P e5482093ec7543069fc6c562145077de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5482093ec7543069fc6c562145077de" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 35179 } } peers { permanent_uuid: "2cb59bb34c2c442da48fb5a3a60ac8b2" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 40307 } } peers { permanent_uuid: "c4726169d9b44c9ead2cc7fcff5d383c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 44759 } }
I20260810 22:38:11.698406 7599 raft_consensus.cc:374] T a46f5c843378412685484ba45615138a P e5482093ec7543069fc6c562145077de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:38:11.698482 7599 raft_consensus.cc:729] T a46f5c843378412685484ba45615138a P e5482093ec7543069fc6c562145077de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5482093ec7543069fc6c562145077de, State: Initialized, Role: FOLLOWER
I20260810 22:38:11.698726 7597 consensus_queue.cc:260] T a46f5c843378412685484ba45615138a P 2cb59bb34c2c442da48fb5a3a60ac8b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5482093ec7543069fc6c562145077de" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 35179 } } peers { permanent_uuid: "2cb59bb34c2c442da48fb5a3a60ac8b2" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 40307 } } peers { permanent_uuid: "c4726169d9b44c9ead2cc7fcff5d383c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 44759 } }
I20260810 22:38:11.698870 7599 consensus_queue.cc:260] T a46f5c843378412685484ba45615138a P e5482093ec7543069fc6c562145077de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5482093ec7543069fc6c562145077de" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 35179 } } peers { permanent_uuid: "2cb59bb34c2c442da48fb5a3a60ac8b2" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 40307 } } peers { permanent_uuid: "c4726169d9b44c9ead2cc7fcff5d383c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 44759 } }
I20260810 22:38:11.699319 7598 ts_tablet_manager.cc:1428] T a46f5c843378412685484ba45615138a P c4726169d9b44c9ead2cc7fcff5d383c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 22:38:11.699582 7441 heartbeater.cc:503] Master 127.16.233.126:46725 was elected leader, sending a full tablet report...
I20260810 22:38:11.699823 7599 ts_tablet_manager.cc:1428] T a46f5c843378412685484ba45615138a P e5482093ec7543069fc6c562145077de: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 22:38:11.699838 7597 ts_tablet_manager.cc:1428] T a46f5c843378412685484ba45615138a P 2cb59bb34c2c442da48fb5a3a60ac8b2: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s
I20260810 22:38:11.700193 7579 heartbeater.cc:503] Master 127.16.233.126:46725 was elected leader, sending a full tablet report...
I20260810 22:38:11.700932 7302 heartbeater.cc:503] Master 127.16.233.126:46725 was elected leader, sending a full tablet report...
W20260810 22:38:11.712858 7303 tablet.cc:2369] T a46f5c843378412685484ba45615138a P c4726169d9b44c9ead2cc7fcff5d383c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:38:11.827348 7604 raft_consensus.cc:482] T a46f5c843378412685484ba45615138a P 2cb59bb34c2c442da48fb5a3a60ac8b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:38:11.827629 7604 raft_consensus.cc:504] T a46f5c843378412685484ba45615138a P 2cb59bb34c2c442da48fb5a3a60ac8b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5482093ec7543069fc6c562145077de" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 35179 } } peers { permanent_uuid: "2cb59bb34c2c442da48fb5a3a60ac8b2" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 40307 } } peers { permanent_uuid: "c4726169d9b44c9ead2cc7fcff5d383c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 44759 } }
I20260810 22:38:11.829075 7604 leader_election.cc:288] T a46f5c843378412685484ba45615138a P 2cb59bb34c2c442da48fb5a3a60ac8b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5482093ec7543069fc6c562145077de (127.16.233.67:35179), c4726169d9b44c9ead2cc7fcff5d383c (127.16.233.65:44759)
I20260810 22:38:11.833874 7596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:11.829773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:46721 (local address 127.16.233.65:44759)
0810 22:38:11.829942 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:11.829946 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:11.830168 (+ 222us) server_negotiation.cc:407] Connection header received
0810 22:38:11.830284 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:11.830287 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:11.830350 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:11.830426 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:38:11.830812 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:11.831375 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:11.832452 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:11.833147 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:11.833278 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:11.833361 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:11.833638 (+ 277us) server_negotiation.cc:299] Negotiation successful
0810 22:38:11.833732 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260810 22:38:11.833914 7443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:11.829302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.67:35179 (local address 127.16.233.66:34843)
0810 22:38:11.829445 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:11.829461 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 22:38:11.829568 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:38:11.829931 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:38:11.829934 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:38:11.829958 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:38:11.830081 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:11.830087 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:11.831083 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:11.831088 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:11.832474 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:11.832489 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:11.833493 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:11.833495 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:11.833619 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:11.833644 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 22:38:11.833714 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 22:38:11.833781 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260810 22:38:11.834307 7606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:11.829580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.65:44759 (local address 127.16.233.66:46721)
0810 22:38:11.830046 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:11.830063 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 22:38:11.830183 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:38:11.830467 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:38:11.830470 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:38:11.830492 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:38:11.830654 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:11.830673 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:11.831493 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:11.831498 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:11.832311 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:11.832320 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:11.833290 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:11.833296 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:11.833434 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:11.833475 (+ 41us) client_negotiation.cc:769] Sending connection context
0810 22:38:11.833571 (+ 96us) client_negotiation.cc:240] Negotiation successful
0810 22:38:11.833665 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":156,"threads_started":1}
I20260810 22:38:11.834316 7592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:11.829469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.233.66:34843 (local address 127.16.233.67:35179)
0810 22:38:11.829608 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:11.829613 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:11.829631 (+ 18us) server_negotiation.cc:407] Connection header received
0810 22:38:11.829699 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:11.829702 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:11.829754 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:11.829872 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:38:11.830257 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:11.830952 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:11.832626 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:11.833267 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:11.833368 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:11.833422 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:11.834088 (+ 666us) server_negotiation.cc:299] Negotiation successful
0810 22:38:11.834183 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 22:38:11.835044 7529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a46f5c843378412685484ba45615138a" candidate_uuid: "2cb59bb34c2c442da48fb5a3a60ac8b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5482093ec7543069fc6c562145077de" is_pre_election: true
I20260810 22:38:11.835111 7254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a46f5c843378412685484ba45615138a" candidate_uuid: "2cb59bb34c2c442da48fb5a3a60ac8b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4726169d9b44c9ead2cc7fcff5d383c" is_pre_election: true
I20260810 22:38:11.835421 7529 raft_consensus.cc:2454] T a46f5c843378412685484ba45615138a P e5482093ec7543069fc6c562145077de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cb59bb34c2c442da48fb5a3a60ac8b2 in term 0.
I20260810 22:38:11.835479 7254 raft_consensus.cc:2454] T a46f5c843378412685484ba45615138a P c4726169d9b44c9ead2cc7fcff5d383c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cb59bb34c2c442da48fb5a3a60ac8b2 in term 0.
I20260810 22:38:11.835983 7323 leader_election.cc:302] T a46f5c843378412685484ba45615138a P 2cb59bb34c2c442da48fb5a3a60ac8b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cb59bb34c2c442da48fb5a3a60ac8b2, e5482093ec7543069fc6c562145077de; no voters:
I20260810 22:38:11.836297 7604 raft_consensus.cc:2781] T a46f5c843378412685484ba45615138a P 2cb59bb34c2c442da48fb5a3a60ac8b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:38:11.836405 7604 raft_consensus.cc:482] T a46f5c843378412685484ba45615138a P 2cb59bb34c2c442da48fb5a3a60ac8b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:38:11.836453 7604 raft_consensus.cc:3037] T a46f5c843378412685484ba45615138a P 2cb59bb34c2c442da48fb5a3a60ac8b2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:38:11.837622 7604 raft_consensus.cc:504] T a46f5c843378412685484ba45615138a P 2cb59bb34c2c442da48fb5a3a60ac8b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5482093ec7543069fc6c562145077de" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 35179 } } peers { permanent_uuid: "2cb59bb34c2c442da48fb5a3a60ac8b2" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 40307 } } peers { permanent_uuid: "c4726169d9b44c9ead2cc7fcff5d383c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 44759 } }
I20260810 22:38:11.838042 7604 leader_election.cc:288] T a46f5c843378412685484ba45615138a P 2cb59bb34c2c442da48fb5a3a60ac8b2 [CANDIDATE]: Term 1 election: Requested vote from peers e5482093ec7543069fc6c562145077de (127.16.233.67:35179), c4726169d9b44c9ead2cc7fcff5d383c (127.16.233.65:44759)
I20260810 22:38:11.838442 7529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a46f5c843378412685484ba45615138a" candidate_uuid: "2cb59bb34c2c442da48fb5a3a60ac8b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5482093ec7543069fc6c562145077de"
I20260810 22:38:11.838511 7254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a46f5c843378412685484ba45615138a" candidate_uuid: "2cb59bb34c2c442da48fb5a3a60ac8b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4726169d9b44c9ead2cc7fcff5d383c"
I20260810 22:38:11.838603 7529 raft_consensus.cc:3037] T a46f5c843378412685484ba45615138a P e5482093ec7543069fc6c562145077de [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:38:11.838670 7254 raft_consensus.cc:3037] T a46f5c843378412685484ba45615138a P c4726169d9b44c9ead2cc7fcff5d383c [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:38:11.839730 7529 raft_consensus.cc:2454] T a46f5c843378412685484ba45615138a P e5482093ec7543069fc6c562145077de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cb59bb34c2c442da48fb5a3a60ac8b2 in term 1.
I20260810 22:38:11.840090 7323 leader_election.cc:302] T a46f5c843378412685484ba45615138a P 2cb59bb34c2c442da48fb5a3a60ac8b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cb59bb34c2c442da48fb5a3a60ac8b2, e5482093ec7543069fc6c562145077de; no voters:
I20260810 22:38:11.840265 7254 raft_consensus.cc:2454] T a46f5c843378412685484ba45615138a P c4726169d9b44c9ead2cc7fcff5d383c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cb59bb34c2c442da48fb5a3a60ac8b2 in term 1.
I20260810 22:38:11.840281 7604 raft_consensus.cc:2781] T a46f5c843378412685484ba45615138a P 2cb59bb34c2c442da48fb5a3a60ac8b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:38:11.840576 7604 raft_consensus.cc:686] T a46f5c843378412685484ba45615138a P 2cb59bb34c2c442da48fb5a3a60ac8b2 [term 1 LEADER]: Becoming Leader. State: Replica: 2cb59bb34c2c442da48fb5a3a60ac8b2, State: Running, Role: LEADER
I20260810 22:38:11.840977 7604 consensus_queue.cc:237] T a46f5c843378412685484ba45615138a P 2cb59bb34c2c442da48fb5a3a60ac8b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5482093ec7543069fc6c562145077de" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 35179 } } peers { permanent_uuid: "2cb59bb34c2c442da48fb5a3a60ac8b2" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 40307 } } peers { permanent_uuid: "c4726169d9b44c9ead2cc7fcff5d383c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 44759 } }
I20260810 22:38:11.844511 7098 catalog_manager.cc:5495] T a46f5c843378412685484ba45615138a P 2cb59bb34c2c442da48fb5a3a60ac8b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2cb59bb34c2c442da48fb5a3a60ac8b2 (127.16.233.66). New cstate: current_term: 1 leader_uuid: "2cb59bb34c2c442da48fb5a3a60ac8b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5482093ec7543069fc6c562145077de" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 35179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2cb59bb34c2c442da48fb5a3a60ac8b2" member_type: VOTER last_known_addr { host: "127.16.233.66" port: 40307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4726169d9b44c9ead2cc7fcff5d383c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 44759 } health_report { overall_health: UNKNOWN } } }
I20260810 22:38:11.867470 7595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:11.862518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58638 (local address 127.16.233.66:40307)
0810 22:38:11.862677 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:11.862682 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:11.862698 (+ 16us) server_negotiation.cc:407] Connection header received
0810 22:38:11.862823 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:11.862827 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:11.862913 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:11.863037 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:38:11.863286 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:11.863832 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:11.865266 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:11.865581 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:11.865669 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:11.866065 (+ 396us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:38:11.866663 (+ 598us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:38:11.866778 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:11.867225 (+ 447us) server_negotiation.cc:299] Negotiation successful
0810 22:38:11.867285 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260810 22:38:11.869360 7580 tablet.cc:2369] T a46f5c843378412685484ba45615138a P e5482093ec7543069fc6c562145077de: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:38:11.875875 7254 raft_consensus.cc:1261] T a46f5c843378412685484ba45615138a P c4726169d9b44c9ead2cc7fcff5d383c [term 1 FOLLOWER]: Refusing update from remote peer 2cb59bb34c2c442da48fb5a3a60ac8b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:38:11.875875 7529 raft_consensus.cc:1261] T a46f5c843378412685484ba45615138a P e5482093ec7543069fc6c562145077de [term 1 FOLLOWER]: Refusing update from remote peer 2cb59bb34c2c442da48fb5a3a60ac8b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:38:11.876710 7607 consensus_queue.cc:1035] T a46f5c843378412685484ba45615138a P 2cb59bb34c2c442da48fb5a3a60ac8b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5482093ec7543069fc6c562145077de" member_type: VOTER last_known_addr { host: "127.16.233.67" port: 35179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:38:11.876950 7604 consensus_queue.cc:1035] T a46f5c843378412685484ba45615138a P 2cb59bb34c2c442da48fb5a3a60ac8b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4726169d9b44c9ead2cc7fcff5d383c" member_type: VOTER last_known_addr { host: "127.16.233.65" port: 44759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:38:11.884222 7610 mvcc.cc:205] Tried to move back new op lower bound from 7317100510715420672 to 7317100510583771136. Current Snapshot: MvccSnapshot[applied={T|T < 7317100510715420672}]
I20260810 22:38:11.885885 7612 mvcc.cc:205] Tried to move back new op lower bound from 7317100510715420672 to 7317100510583771136. Current Snapshot: MvccSnapshot[applied={T|T < 7317100510715420672}]
I20260810 22:38:11.894569 7099 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51266:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:38:11.894784 7099 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51266:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:38:11.897742 7099 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 76ca050f630a47a1b919fc55e1f25183: Sending DeleteTablet for 3 replicas of tablet a46f5c843378412685484ba45615138a
I20260810 22:38:11.898784 7371 tablet_service.cc:1514] Processing DeleteTablet for tablet a46f5c843378412685484ba45615138a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:38:11 UTC) from {username='slave'} at 127.0.0.1:58632
I20260810 22:38:11.898916 7230 tablet_service.cc:1514] Processing DeleteTablet for tablet a46f5c843378412685484ba45615138a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:38:11 UTC) from {username='slave'} at 127.0.0.1:37466
I20260810 22:38:11.898928 7508 tablet_service.cc:1514] Processing DeleteTablet for tablet a46f5c843378412685484ba45615138a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:38:11 UTC) from {username='slave'} at 127.0.0.1:51422
I20260810 22:38:11.899516 7621 tablet_replica.cc:330] T a46f5c843378412685484ba45615138a P e5482093ec7543069fc6c562145077de: stopping tablet replica
I20260810 22:38:11.899910 7621 raft_consensus.cc:2229] T a46f5c843378412685484ba45615138a P e5482093ec7543069fc6c562145077de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:38:11.900182 7619 tablet_replica.cc:330] T a46f5c843378412685484ba45615138a P 2cb59bb34c2c442da48fb5a3a60ac8b2: stopping tablet replica
I20260810 22:38:11.900272 7621 raft_consensus.cc:2258] T a46f5c843378412685484ba45615138a P e5482093ec7543069fc6c562145077de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:38:11.900532 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 7166
I20260810 22:38:11.905625 7619 raft_consensus.cc:2229] T a46f5c843378412685484ba45615138a P 2cb59bb34c2c442da48fb5a3a60ac8b2 [term 1 LEADER]: Raft consensus shutting down.
I20260810 22:38:11.905957 7621 ts_tablet_manager.cc:1905] T a46f5c843378412685484ba45615138a P e5482093ec7543069fc6c562145077de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:38:11.906190 7619 raft_consensus.cc:2258] T a46f5c843378412685484ba45615138a P 2cb59bb34c2c442da48fb5a3a60ac8b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:38:11.909448 7621 ts_tablet_manager.cc:1918] T a46f5c843378412685484ba45615138a P e5482093ec7543069fc6c562145077de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:38:11.909579 7621 log.cc:1196] T a46f5c843378412685484ba45615138a P e5482093ec7543069fc6c562145077de: Deleting WAL directory at /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786401215402979-17317-0/minicluster-data/ts-2/wal/wals/a46f5c843378412685484ba45615138a
I20260810 22:38:11.909952 7621 ts_tablet_manager.cc:1939] T a46f5c843378412685484ba45615138a P e5482093ec7543069fc6c562145077de: Deleting consensus metadata
I20260810 22:38:11.911513 7084 catalog_manager.cc:4841] TS e5482093ec7543069fc6c562145077de (127.16.233.67:35179): tablet a46f5c843378412685484ba45615138a (table test-table [id=a0186666ebd4410fa6828259272f33dc]) successfully deleted
I20260810 22:38:11.913040 7619 ts_tablet_manager.cc:1905] T a46f5c843378412685484ba45615138a P 2cb59bb34c2c442da48fb5a3a60ac8b2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 22:38:11.915104 7084 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.233.65:44759 (error 108)
I20260810 22:38:11.915262 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 7305
I20260810 22:38:11.922227 7593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:11.921852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.65:44759 (local address 127.0.0.1:37480)
0810 22:38:11.922015 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:11.922110 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.65:44759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260810 22:38:11.922986 7084 catalog_manager.cc:4584] TS c4726169d9b44c9ead2cc7fcff5d383c (127.16.233.65:44759): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a46f5c843378412685484ba45615138a: Network error: Client connection negotiation failed: client connection to 127.16.233.65:44759: connect: Connection refused (error 111)
I20260810 22:38:11.928606 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 7444
I20260810 22:38:11.933128 7593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:11.932681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.66:40307 (local address 127.0.0.1:58642)
0810 22:38:11.932911 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:11.932991 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.66:40307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260810 22:38:11.937506 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 7065
2026-08-10T22:38:11Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1206 ms)
[----------] 6 tests from EncryptionPolicyTest (12249 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786401215402979-17317-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:38:12 dist-test-slave-77v9 krb5kdc[7628](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:38:12 dist-test-slave-77v9 krb5kdc[7628](info): set up 2 sockets
Aug 10 22:38:12 dist-test-slave-77v9 krb5kdc[7628](info): commencing operation
krb5kdc: starting...
W20260810 22:38:13.090658 17317 mini_kdc.cc:121] Time spent starting KDC: real 1.128s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Aug 10 22:38:13 dist-test-slave-77v9 krb5kdc[7628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:38:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:38:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786401215402979-17317-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786401215402979-17317-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 22:38:13.236765 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786401215402979-17317-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786401215402979-17317-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786401215402979-17317-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46267
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786401215402979-17317-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46267
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:38:13.359477 7641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:38:13.359845 7641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:38:13.359927 7641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:38:13.364176 7641 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:38:13.364302 7641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:38:13.364346 7641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:38:13.364403 7641 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:38:13.364442 7641 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:38:13.370273 7641 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786401215402979-17317-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786401215402979-17317-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46267
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786401215402979-17317-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786401215402979-17317-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:38:13.371542 7641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:38:13.372649 7641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:38:13.381750 7646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:13.381778 7649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:13.381778 7647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:38:13.382023 7641 server_base.cc:1034] running on GCE node
I20260810 22:38:13.382829 7641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:38:13.384042 7641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:38:13.385248 7641 hybrid_clock.cc:648] HybridClock initialized: now 1786401493385228 us; error 29 us; skew 500 ppm
Aug 10 22:38:13 dist-test-slave-77v9 krb5kdc[7628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401493, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:38:13.388482 7641 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 22:38:13.389554 7641 webserver.cc:460] Webserver started at http://127.0.0.1:42443/ using document root <none> and password file <none>
I20260810 22:38:13.390127 7641 fs_manager.cc:359] Metadata directory not provided
I20260810 22:38:13.390190 7641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:38:13.390370 7641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:38:13.392562 7641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786401215402979-17317-0/minicluster-data/master-0/data/instance:
uuid: "48cc70968e7b46e3a149e56c38fcb679"
format_stamp: "Formatted at 2026-08-10 22:38:13 on dist-test-slave-77v9"
server_key: "ca8cd74f048bde4c86503c1563a05d4b"
server_key_iv: "cd404321526540f8e225525acd4a2e63"
server_key_version: "encryptionkey@0"
I20260810 22:38:13.393219 7641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786401215402979-17317-0/minicluster-data/master-0/wal/instance:
uuid: "48cc70968e7b46e3a149e56c38fcb679"
format_stamp: "Formatted at 2026-08-10 22:38:13 on dist-test-slave-77v9"
server_key: "ca8cd74f048bde4c86503c1563a05d4b"
server_key_iv: "cd404321526540f8e225525acd4a2e63"
server_key_version: "encryptionkey@0"
I20260810 22:38:13.397300 7641 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 22:38:13.399919 7657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:13.401366 7641 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:38:13.401484 7641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786401215402979-17317-0/minicluster-data/master-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786401215402979-17317-0/minicluster-data/master-0/wal
uuid: "48cc70968e7b46e3a149e56c38fcb679"
format_stamp: "Formatted at 2026-08-10 22:38:13 on dist-test-slave-77v9"
server_key: "ca8cd74f048bde4c86503c1563a05d4b"
server_key_iv: "cd404321526540f8e225525acd4a2e63"
server_key_version: "encryptionkey@0"
I20260810 22:38:13.401587 7641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786401215402979-17317-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786401215402979-17317-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786401215402979-17317-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:38:13.431089 7641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:38:13.436100 7641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:38:13.436367 7641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:38:13.445307 7641 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46267
I20260810 22:38:13.445382 7712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46267 every 8 connection(s)
I20260810 22:38:13.446444 7641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
I20260810 22:38:13.449592 7713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:38:13.455549 7713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48cc70968e7b46e3a149e56c38fcb679: Bootstrap starting.
I20260810 22:38:13.456359 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 7641
I20260810 22:38:13.456514 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786401215402979-17317-0/minicluster-data/master-0/wal/instance
I20260810 22:38:13.456844 17317 external_mini_cluster.cc:1428] Setting key e0a6fd652ea1f466ac7a163f498a7761
I20260810 22:38:13.458767 7713 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48cc70968e7b46e3a149e56c38fcb679: Neither blocks nor log segments found. Creating new log.
I20260810 22:38:13.459537 7713 log.cc:824] T 00000000000000000000000000000000 P 48cc70968e7b46e3a149e56c38fcb679: Log is configured to *not* fsync() on all Append() calls
I20260810 22:38:13.461921 7713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48cc70968e7b46e3a149e56c38fcb679: No bootstrap required, opened a new log
Aug 10 22:38:13 dist-test-slave-77v9 krb5kdc[7628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 22:38:13.466964 7713 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48cc70968e7b46e3a149e56c38fcb679 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48cc70968e7b46e3a149e56c38fcb679" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46267 } }
I20260810 22:38:13.467249 7713 raft_consensus.cc:374] T 00000000000000000000000000000000 P 48cc70968e7b46e3a149e56c38fcb679 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:38:13.467330 7713 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48cc70968e7b46e3a149e56c38fcb679 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48cc70968e7b46e3a149e56c38fcb679, State: Initialized, Role: FOLLOWER
I20260810 22:38:13.467865 7713 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48cc70968e7b46e3a149e56c38fcb679 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48cc70968e7b46e3a149e56c38fcb679" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46267 } }
I20260810 22:38:13.468027 7713 raft_consensus.cc:388] T 00000000000000000000000000000000 P 48cc70968e7b46e3a149e56c38fcb679 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:38:13.468120 7713 raft_consensus.cc:482] T 00000000000000000000000000000000 P 48cc70968e7b46e3a149e56c38fcb679 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:38:13.468227 7713 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48cc70968e7b46e3a149e56c38fcb679 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:38:13.469312 7713 raft_consensus.cc:504] T 00000000000000000000000000000000 P 48cc70968e7b46e3a149e56c38fcb679 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48cc70968e7b46e3a149e56c38fcb679" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46267 } }
I20260810 22:38:13.469681 7713 leader_election.cc:302] T 00000000000000000000000000000000 P 48cc70968e7b46e3a149e56c38fcb679 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48cc70968e7b46e3a149e56c38fcb679; no voters:
I20260810 22:38:13.470052 7713 leader_election.cc:288] T 00000000000000000000000000000000 P 48cc70968e7b46e3a149e56c38fcb679 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:38:13.470134 7718 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 48cc70968e7b46e3a149e56c38fcb679 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:38:13.470337 7718 raft_consensus.cc:686] T 00000000000000000000000000000000 P 48cc70968e7b46e3a149e56c38fcb679 [term 1 LEADER]: Becoming Leader. State: Replica: 48cc70968e7b46e3a149e56c38fcb679, State: Running, Role: LEADER
I20260810 22:38:13.470649 7718 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48cc70968e7b46e3a149e56c38fcb679 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48cc70968e7b46e3a149e56c38fcb679" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46267 } }
I20260810 22:38:13.471446 7713 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48cc70968e7b46e3a149e56c38fcb679 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:38:13.472384 7719 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48cc70968e7b46e3a149e56c38fcb679 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "48cc70968e7b46e3a149e56c38fcb679" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48cc70968e7b46e3a149e56c38fcb679" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46267 } } }
I20260810 22:38:13.472536 7719 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48cc70968e7b46e3a149e56c38fcb679 [sys.catalog]: This master's current role is: LEADER
I20260810 22:38:13.472774 7720 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48cc70968e7b46e3a149e56c38fcb679 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48cc70968e7b46e3a149e56c38fcb679. Latest consensus state: current_term: 1 leader_uuid: "48cc70968e7b46e3a149e56c38fcb679" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48cc70968e7b46e3a149e56c38fcb679" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46267 } } }
I20260810 22:38:13.472911 7720 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48cc70968e7b46e3a149e56c38fcb679 [sys.catalog]: This master's current role is: LEADER
I20260810 22:38:13.473353 7727 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:38:13.474893 7716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:13.458239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52907 (local address 127.0.0.1:46267)
0810 22:38:13.458692 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:13.458707 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:13.458745 (+ 38us) server_negotiation.cc:407] Connection header received
0810 22:38:13.459472 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:13.459490 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:13.459927 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:13.460327 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:13.461245 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:13.462334 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:13.463498 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:13.463983 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:13.464108 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:13.467569 (+ 3461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:13.467575 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:13.467589 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:38:13.467626 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:13.469776 (+ 2150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:13.470573 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:13.470580 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:13.470587 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:13.470753 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:13.471179 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:13.471186 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:13.471189 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:13.471887 (+ 698us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:13.472152 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:13.473483 (+ 1331us) server_negotiation.cc:299] Negotiation successful
0810 22:38:13.473745 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":297,"thread_start_us":151,"threads_started":1}
I20260810 22:38:13.477550 7727 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:38:13.478204 17317 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 22:38:13.486284 7727 catalog_manager.cc:1347] Generated new cluster ID: d5239ea71cbc44ed8aa36a5bc0a3abdb
I20260810 22:38:13.486382 7727 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:38:13.491559 7716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:13.480645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50073 (local address 127.0.0.1:46267)
0810 22:38:13.480804 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:13.480834 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:13.482663 (+ 1829us) server_negotiation.cc:407] Connection header received
0810 22:38:13.482796 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:13.482799 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:13.482853 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:13.482947 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:13.483934 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:13.484612 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:13.485811 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:13.486161 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:13.486270 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:13.488030 (+ 1760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:13.488036 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:13.488044 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:38:13.488069 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:13.489638 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:13.490159 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:13.490165 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:13.490171 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:13.490249 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:13.490595 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:13.490598 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:13.490600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:13.490785 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:13.490975 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:13.491255 (+ 280us) server_negotiation.cc:299] Negotiation successful
0810 22:38:13.491357 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 22:38:13.502913 7727 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:38:13.503918 7727 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:38:13.509479 7727 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 48cc70968e7b46e3a149e56c38fcb679: Generated new TSK 0
I20260810 22:38:13.510389 7727 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:38:13.517160 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 7641
W20260810 22:38:13.526093 7624 connection.cc:535] client connection to 127.0.0.1:46267 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46267): Connection reset by peer (error 104)
2026-08-10T22:38:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1578 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786401215402979-17317-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:38:13 dist-test-slave-77v9 krb5kdc[7746](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:38:13 dist-test-slave-77v9 krb5kdc[7746](info): set up 2 sockets
Aug 10 22:38:13 dist-test-slave-77v9 krb5kdc[7746](info): commencing operation
krb5kdc: starting...
W20260810 22:38:14.646718 17317 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Aug 10 22:38:14 dist-test-slave-77v9 krb5kdc[7746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:38:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:38:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786401215402979-17317-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786401215402979-17317-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 22:38:14.794574 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786401215402979-17317-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786401215402979-17317-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786401215402979-17317-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35091
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786401215402979-17317-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35091
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:38:14.913822 7760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:38:14.914187 7760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:38:14.914274 7760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:38:14.918594 7760 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:38:14.918718 7760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:38:14.918759 7760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:38:14.918802 7760 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:38:14.918843 7760 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:38:14.925019 7760 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786401215402979-17317-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786401215402979-17317-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35091
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786401215402979-17317-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7760
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786401215402979-17317-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:38:14.926311 7760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:38:14.927461 7760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:38:14.936761 7766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:14.936764 7768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:14.936861 7765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:38:14.936851 7760 server_base.cc:1034] running on GCE node
I20260810 22:38:14.937736 7760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:38:14.939186 7760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:38:14.940398 7760 hybrid_clock.cc:648] HybridClock initialized: now 1786401494940375 us; error 55 us; skew 500 ppm
Aug 10 22:38:14 dist-test-slave-77v9 krb5kdc[7746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401494, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:38:14.943802 7760 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 22:38:14.944953 7760 webserver.cc:460] Webserver started at http://127.0.0.1:35635/ using document root <none> and password file <none>
I20260810 22:38:14.945576 7760 fs_manager.cc:359] Metadata directory not provided
I20260810 22:38:14.945678 7760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:38:14.945960 7760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:38:14.948258 7760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786401215402979-17317-0/minicluster-data/master-0/data/instance:
uuid: "51e2bac8fb7c494eb60dedc1fb644667"
format_stamp: "Formatted at 2026-08-10 22:38:14 on dist-test-slave-77v9"
server_key: "31f282335b5d07d4aa3383b83b9ebfd1"
server_key_iv: "481817d4ccf09db5794dab038023f15f"
server_key_version: "encryptionkey@0"
I20260810 22:38:14.949007 7760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786401215402979-17317-0/minicluster-data/master-0/wal/instance:
uuid: "51e2bac8fb7c494eb60dedc1fb644667"
format_stamp: "Formatted at 2026-08-10 22:38:14 on dist-test-slave-77v9"
server_key: "31f282335b5d07d4aa3383b83b9ebfd1"
server_key_iv: "481817d4ccf09db5794dab038023f15f"
server_key_version: "encryptionkey@0"
I20260810 22:38:14.953518 7760 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 22:38:14.956278 7777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:14.957821 7760 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:38:14.958016 7760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786401215402979-17317-0/minicluster-data/master-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786401215402979-17317-0/minicluster-data/master-0/wal
uuid: "51e2bac8fb7c494eb60dedc1fb644667"
format_stamp: "Formatted at 2026-08-10 22:38:14 on dist-test-slave-77v9"
server_key: "31f282335b5d07d4aa3383b83b9ebfd1"
server_key_iv: "481817d4ccf09db5794dab038023f15f"
server_key_version: "encryptionkey@0"
I20260810 22:38:14.958156 7760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786401215402979-17317-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786401215402979-17317-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786401215402979-17317-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:38:14.995451 7760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:38:14.998889 7760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:38:14.999146 7760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:38:15.007898 7760 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35091
I20260810 22:38:15.007930 7833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35091 every 8 connection(s)
I20260810 22:38:15.009214 7760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
I20260810 22:38:15.012223 7834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:38:15.014237 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 7760
I20260810 22:38:15.014394 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786401215402979-17317-0/minicluster-data/master-0/wal/instance
I20260810 22:38:15.014727 17317 external_mini_cluster.cc:1428] Setting key 1bd8a81971772dfe8019a99211b495fb
I20260810 22:38:15.019816 7834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51e2bac8fb7c494eb60dedc1fb644667: Bootstrap starting.
I20260810 22:38:15.022703 7834 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51e2bac8fb7c494eb60dedc1fb644667: Neither blocks nor log segments found. Creating new log.
I20260810 22:38:15.023746 7834 log.cc:824] T 00000000000000000000000000000000 P 51e2bac8fb7c494eb60dedc1fb644667: Log is configured to *not* fsync() on all Append() calls
I20260810 22:38:15.025714 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:15.016358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43573 (local address 127.0.0.1:35091)
0810 22:38:15.016889 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:15.016910 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:15.016956 (+ 46us) server_negotiation.cc:407] Connection header received
0810 22:38:15.017792 (+ 836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:15.017819 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:15.018297 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:15.018711 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:15.019147 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:15.020577 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:15.021606 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:15.022276 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:15.022458 (+ 182us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:15.023507 (+ 1049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:15.023517 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:15.023537 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:38:15.024089 (+ 552us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:15.024172 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:38:15.024206 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:15.024378 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:15.024577 (+ 199us) server_negotiation.cc:299] Negotiation successful
0810 22:38:15.024942 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":141,"threads_started":1}
I20260810 22:38:15.026078 7834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51e2bac8fb7c494eb60dedc1fb644667: No bootstrap required, opened a new log
I20260810 22:38:15.028988 17317 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 22:38:15.030366 7834 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51e2bac8fb7c494eb60dedc1fb644667 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51e2bac8fb7c494eb60dedc1fb644667" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35091 } }
I20260810 22:38:15.030666 7834 raft_consensus.cc:374] T 00000000000000000000000000000000 P 51e2bac8fb7c494eb60dedc1fb644667 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:38:15.030743 7834 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51e2bac8fb7c494eb60dedc1fb644667 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51e2bac8fb7c494eb60dedc1fb644667, State: Initialized, Role: FOLLOWER
I20260810 22:38:15.031296 7834 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51e2bac8fb7c494eb60dedc1fb644667 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51e2bac8fb7c494eb60dedc1fb644667" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35091 } }
I20260810 22:38:15.031541 7834 raft_consensus.cc:388] T 00000000000000000000000000000000 P 51e2bac8fb7c494eb60dedc1fb644667 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:38:15.031629 7834 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51e2bac8fb7c494eb60dedc1fb644667 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:38:15.031742 7834 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51e2bac8fb7c494eb60dedc1fb644667 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:38:15.033208 7834 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51e2bac8fb7c494eb60dedc1fb644667 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51e2bac8fb7c494eb60dedc1fb644667" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35091 } }
I20260810 22:38:15.033620 7834 leader_election.cc:302] T 00000000000000000000000000000000 P 51e2bac8fb7c494eb60dedc1fb644667 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51e2bac8fb7c494eb60dedc1fb644667; no voters:
I20260810 22:38:15.034014 7834 leader_election.cc:288] T 00000000000000000000000000000000 P 51e2bac8fb7c494eb60dedc1fb644667 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:38:15.034144 7846 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51e2bac8fb7c494eb60dedc1fb644667 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:38:15.034433 7846 raft_consensus.cc:686] T 00000000000000000000000000000000 P 51e2bac8fb7c494eb60dedc1fb644667 [term 1 LEADER]: Becoming Leader. State: Replica: 51e2bac8fb7c494eb60dedc1fb644667, State: Running, Role: LEADER
I20260810 22:38:15.034813 7846 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51e2bac8fb7c494eb60dedc1fb644667 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51e2bac8fb7c494eb60dedc1fb644667" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35091 } }
I20260810 22:38:15.035547 7834 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51e2bac8fb7c494eb60dedc1fb644667 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:38:15.036309 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:15.031306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44033 (local address 127.0.0.1:35091)
0810 22:38:15.031469 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:15.031474 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:15.031673 (+ 199us) server_negotiation.cc:407] Connection header received
0810 22:38:15.031839 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:15.031844 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:15.031914 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:15.032006 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:15.032273 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:15.033075 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:15.034014 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:15.034396 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:15.034496 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:15.035580 (+ 1084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:15.035584 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:15.035587 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:38:15.035603 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:15.035633 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:38:15.035649 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:15.035731 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:15.036118 (+ 387us) server_negotiation.cc:299] Negotiation successful
0810 22:38:15.036154 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":52}
I20260810 22:38:15.036585 7849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51e2bac8fb7c494eb60dedc1fb644667 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51e2bac8fb7c494eb60dedc1fb644667. Latest consensus state: current_term: 1 leader_uuid: "51e2bac8fb7c494eb60dedc1fb644667" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51e2bac8fb7c494eb60dedc1fb644667" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35091 } } }
I20260810 22:38:15.036722 7849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51e2bac8fb7c494eb60dedc1fb644667 [sys.catalog]: This master's current role is: LEADER
I20260810 22:38:15.036973 7847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51e2bac8fb7c494eb60dedc1fb644667 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "51e2bac8fb7c494eb60dedc1fb644667" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51e2bac8fb7c494eb60dedc1fb644667" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35091 } } }
I20260810 22:38:15.037073 7847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51e2bac8fb7c494eb60dedc1fb644667 [sys.catalog]: This master's current role is: LEADER
I20260810 22:38:15.037510 7856 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:38:15.040719 7856 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:38:15.047025 7856 catalog_manager.cc:1347] Generated new cluster ID: 1949d4d879d24d0e9dcca12b43b93ddb
I20260810 22:38:15.047132 7856 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:38:15.058885 7856 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:38:15.060318 7856 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:38:15.081781 7856 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 51e2bac8fb7c494eb60dedc1fb644667: Generated new TSK 0
I20260810 22:38:15.082609 7856 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:38:15.093560 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 7760
W20260810 22:38:15.101513 7743 connection.cc:535] client connection to 127.0.0.1:35091 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35091): Connection reset by peer (error 104)
2026-08-10T22:38:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1574 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786401215402979-17317-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:38:15 dist-test-slave-77v9 krb5kdc[7868](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:38:15 dist-test-slave-77v9 krb5kdc[7868](info): set up 2 sockets
Aug 10 22:38:15 dist-test-slave-77v9 krb5kdc[7868](info): commencing operation
krb5kdc: starting...
W20260810 22:38:16.270684 17317 mini_kdc.cc:121] Time spent starting KDC: real 1.154s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Aug 10 22:38:16 dist-test-slave-77v9 krb5kdc[7868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401496, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:38:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:38:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786401215402979-17317-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786401215402979-17317-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 22:38:16.421720 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786401215402979-17317-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786401215402979-17317-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786401215402979-17317-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41389
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786401215402979-17317-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41389
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:38:16.550941 7882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:38:16.551348 7882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:38:16.551446 7882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:38:16.556020 7882 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:38:16.556154 7882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:38:16.556193 7882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:38:16.556233 7882 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:38:16.556270 7882 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:38:16.562288 7882 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786401215402979-17317-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786401215402979-17317-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41389
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786401215402979-17317-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7882
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786401215402979-17317-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:38:16.563552 7882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:38:16.564702 7882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:38:16.574206 7888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:38:16.574473 7882 server_base.cc:1034] running on GCE node
W20260810 22:38:16.574232 7892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:16.574203 7889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:38:16.575269 7882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:38:16.576618 7882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:38:16.577769 7882 hybrid_clock.cc:648] HybridClock initialized: now 1786401496577746 us; error 44 us; skew 500 ppm
Aug 10 22:38:16 dist-test-slave-77v9 krb5kdc[7868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401496, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:38:16.581282 7882 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 22:38:16.582378 7882 webserver.cc:460] Webserver started at http://127.0.0.1:33167/ using document root <none> and password file <none>
I20260810 22:38:16.583009 7882 fs_manager.cc:359] Metadata directory not provided
I20260810 22:38:16.583081 7882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:38:16.583310 7882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:38:16.585690 7882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786401215402979-17317-0/minicluster-data/master-0/data/instance:
uuid: "f85907317b974562a61431d2d9abb58e"
format_stamp: "Formatted at 2026-08-10 22:38:16 on dist-test-slave-77v9"
server_key: "2602c340d4fd5353c45689d67b1b468a"
server_key_iv: "7f3cf7fd800a23d97c7cdc9229c11944"
server_key_version: "encryptionkey@0"
I20260810 22:38:16.586339 7882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786401215402979-17317-0/minicluster-data/master-0/wal/instance:
uuid: "f85907317b974562a61431d2d9abb58e"
format_stamp: "Formatted at 2026-08-10 22:38:16 on dist-test-slave-77v9"
server_key: "2602c340d4fd5353c45689d67b1b468a"
server_key_iv: "7f3cf7fd800a23d97c7cdc9229c11944"
server_key_version: "encryptionkey@0"
I20260810 22:38:16.590780 7882 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 22:38:16.593770 7899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:16.595261 7882 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 22:38:16.595455 7882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786401215402979-17317-0/minicluster-data/master-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786401215402979-17317-0/minicluster-data/master-0/wal
uuid: "f85907317b974562a61431d2d9abb58e"
format_stamp: "Formatted at 2026-08-10 22:38:16 on dist-test-slave-77v9"
server_key: "2602c340d4fd5353c45689d67b1b468a"
server_key_iv: "7f3cf7fd800a23d97c7cdc9229c11944"
server_key_version: "encryptionkey@0"
I20260810 22:38:16.595603 7882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786401215402979-17317-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786401215402979-17317-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786401215402979-17317-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:38:16.600886 7882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:38:16.604344 7882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:38:16.604591 7882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:38:16.613463 7882 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41389
I20260810 22:38:16.613492 7954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41389 every 8 connection(s)
I20260810 22:38:16.614625 7882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
I20260810 22:38:16.618346 7955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:38:16.621094 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 7882
I20260810 22:38:16.621271 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786401215402979-17317-0/minicluster-data/master-0/wal/instance
I20260810 22:38:16.621567 17317 external_mini_cluster.cc:1428] Setting key 0c28e96afed77979ee7ca3fc51316ca0
I20260810 22:38:16.625787 7955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f85907317b974562a61431d2d9abb58e: Bootstrap starting.
I20260810 22:38:16.627758 7958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:16.623374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40653 (local address 127.0.0.1:41389)
0810 22:38:16.623869 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:16.623887 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:16.623926 (+ 39us) server_negotiation.cc:407] Connection header received
0810 22:38:16.624604 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:16.624622 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:16.625001 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:16.625304 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:16.626372 (+ 1068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:16.626377 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:16.626390 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:38:16.626401 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:16.626473 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:38:16.626494 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:16.626610 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:16.626748 (+ 138us) server_negotiation.cc:299] Negotiation successful
0810 22:38:16.626931 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":162,"threads_started":1}
I20260810 22:38:16.628942 7955 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f85907317b974562a61431d2d9abb58e: Neither blocks nor log segments found. Creating new log.
I20260810 22:38:16.629918 7955 log.cc:824] T 00000000000000000000000000000000 P f85907317b974562a61431d2d9abb58e: Log is configured to *not* fsync() on all Append() calls
I20260810 22:38:16.631158 17317 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 22:38:16.632268 7955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f85907317b974562a61431d2d9abb58e: No bootstrap required, opened a new log
I20260810 22:38:16.635689 7958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:16.633750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58399 (local address 127.0.0.1:41389)
0810 22:38:16.633902 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:16.633908 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:16.634189 (+ 281us) server_negotiation.cc:407] Connection header received
0810 22:38:16.634298 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:16.634301 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:16.634344 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:16.634445 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:16.635242 (+ 797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:16.635246 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:16.635250 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:38:16.635264 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:16.635297 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:38:16.635315 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:16.635433 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:16.635495 (+ 62us) server_negotiation.cc:299] Negotiation successful
0810 22:38:16.635540 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 22:38:16.637620 7955 raft_consensus.cc:348] T 00000000000000000000000000000000 P f85907317b974562a61431d2d9abb58e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f85907317b974562a61431d2d9abb58e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41389 } }
I20260810 22:38:16.637915 7955 raft_consensus.cc:374] T 00000000000000000000000000000000 P f85907317b974562a61431d2d9abb58e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:38:16.637997 7955 raft_consensus.cc:729] T 00000000000000000000000000000000 P f85907317b974562a61431d2d9abb58e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f85907317b974562a61431d2d9abb58e, State: Initialized, Role: FOLLOWER
I20260810 22:38:16.638499 7955 consensus_queue.cc:260] T 00000000000000000000000000000000 P f85907317b974562a61431d2d9abb58e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f85907317b974562a61431d2d9abb58e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41389 } }
I20260810 22:38:16.638670 7955 raft_consensus.cc:388] T 00000000000000000000000000000000 P f85907317b974562a61431d2d9abb58e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:38:16.638770 7955 raft_consensus.cc:482] T 00000000000000000000000000000000 P f85907317b974562a61431d2d9abb58e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:38:16.638904 7955 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f85907317b974562a61431d2d9abb58e [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:38:16.640237 7955 raft_consensus.cc:504] T 00000000000000000000000000000000 P f85907317b974562a61431d2d9abb58e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f85907317b974562a61431d2d9abb58e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41389 } }
I20260810 22:38:16.640657 7955 leader_election.cc:302] T 00000000000000000000000000000000 P f85907317b974562a61431d2d9abb58e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f85907317b974562a61431d2d9abb58e; no voters:
I20260810 22:38:16.641103 7955 leader_election.cc:288] T 00000000000000000000000000000000 P f85907317b974562a61431d2d9abb58e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:38:16.641310 7965 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f85907317b974562a61431d2d9abb58e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:38:16.641590 7965 raft_consensus.cc:686] T 00000000000000000000000000000000 P f85907317b974562a61431d2d9abb58e [term 1 LEADER]: Becoming Leader. State: Replica: f85907317b974562a61431d2d9abb58e, State: Running, Role: LEADER
I20260810 22:38:16.641978 7965 consensus_queue.cc:237] T 00000000000000000000000000000000 P f85907317b974562a61431d2d9abb58e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f85907317b974562a61431d2d9abb58e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41389 } }
I20260810 22:38:16.642581 7955 sys_catalog.cc:564] T 00000000000000000000000000000000 P f85907317b974562a61431d2d9abb58e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:38:16.643844 7967 sys_catalog.cc:455] T 00000000000000000000000000000000 P f85907317b974562a61431d2d9abb58e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f85907317b974562a61431d2d9abb58e. Latest consensus state: current_term: 1 leader_uuid: "f85907317b974562a61431d2d9abb58e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f85907317b974562a61431d2d9abb58e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41389 } } }
I20260810 22:38:16.643998 7967 sys_catalog.cc:458] T 00000000000000000000000000000000 P f85907317b974562a61431d2d9abb58e [sys.catalog]: This master's current role is: LEADER
I20260810 22:38:16.644316 7966 sys_catalog.cc:455] T 00000000000000000000000000000000 P f85907317b974562a61431d2d9abb58e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f85907317b974562a61431d2d9abb58e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f85907317b974562a61431d2d9abb58e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41389 } } }
I20260810 22:38:16.644416 7966 sys_catalog.cc:458] T 00000000000000000000000000000000 P f85907317b974562a61431d2d9abb58e [sys.catalog]: This master's current role is: LEADER
I20260810 22:38:16.644551 7975 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:38:16.647727 7975 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:38:16.654011 7975 catalog_manager.cc:1347] Generated new cluster ID: 60ea4bb0d6894b0c86e82ab94899605f
I20260810 22:38:16.654109 7975 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:38:16.666518 7975 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:38:16.666899 7975 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:38:16.678695 7975 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f85907317b974562a61431d2d9abb58e: Generated new TSK 0
I20260810 22:38:16.679641 7975 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:38:16.694950 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 7882
2026-08-10T22:38:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1603 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786401215402979-17317-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:38:16 dist-test-slave-77v9 krb5kdc[7990](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:38:16 dist-test-slave-77v9 krb5kdc[7990](info): set up 2 sockets
Aug 10 22:38:16 dist-test-slave-77v9 krb5kdc[7990](info): commencing operation
krb5kdc: starting...
W20260810 22:38:17.840306 17317 mini_kdc.cc:121] Time spent starting KDC: real 1.120s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Aug 10 22:38:17 dist-test-slave-77v9 krb5kdc[7990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401497, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:38:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:38:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786401215402979-17317-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786401215402979-17317-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 22:38:17.988888 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786401215402979-17317-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786401215402979-17317-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786401215402979-17317-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45999
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786401215402979-17317-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45999
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:38:18.116717 8004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:38:18.117239 8004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:38:18.117324 8004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:38:18.121665 8004 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:38:18.121807 8004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:38:18.121850 8004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:38:18.121891 8004 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:38:18.121922 8004 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:38:18.127844 8004 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786401215402979-17317-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786401215402979-17317-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786401215402979-17317-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786401215402979-17317-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:38:18.129179 8004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:38:18.130307 8004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:38:18.139643 8011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:18.139645 8013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:18.139655 8010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:38:18.139954 8004 server_base.cc:1034] running on GCE node
I20260810 22:38:18.140726 8004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:38:18.142037 8004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:38:18.143219 8004 hybrid_clock.cc:648] HybridClock initialized: now 1786401498143206 us; error 32 us; skew 500 ppm
Aug 10 22:38:18 dist-test-slave-77v9 krb5kdc[7990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401498, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:38:18.146390 8004 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 22:38:18.147426 8004 webserver.cc:460] Webserver started at http://127.0.0.1:42197/ using document root <none> and password file <none>
I20260810 22:38:18.148115 8004 fs_manager.cc:359] Metadata directory not provided
I20260810 22:38:18.148238 8004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:38:18.148514 8004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:38:18.150846 8004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786401215402979-17317-0/minicluster-data/master-0/data/instance:
uuid: "5476f5718dfb446ebb553d471b180bb9"
format_stamp: "Formatted at 2026-08-10 22:38:18 on dist-test-slave-77v9"
server_key: "13851a72d892c2227e1f4a0b02bff772"
server_key_iv: "eb5d9f73513d4ded9fb39334a2379490"
server_key_version: "encryptionkey@0"
I20260810 22:38:18.151476 8004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786401215402979-17317-0/minicluster-data/master-0/wal/instance:
uuid: "5476f5718dfb446ebb553d471b180bb9"
format_stamp: "Formatted at 2026-08-10 22:38:18 on dist-test-slave-77v9"
server_key: "13851a72d892c2227e1f4a0b02bff772"
server_key_iv: "eb5d9f73513d4ded9fb39334a2379490"
server_key_version: "encryptionkey@0"
I20260810 22:38:18.155737 8004 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260810 22:38:18.158562 8020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:18.160215 8004 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 22:38:18.160398 8004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786401215402979-17317-0/minicluster-data/master-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786401215402979-17317-0/minicluster-data/master-0/wal
uuid: "5476f5718dfb446ebb553d471b180bb9"
format_stamp: "Formatted at 2026-08-10 22:38:18 on dist-test-slave-77v9"
server_key: "13851a72d892c2227e1f4a0b02bff772"
server_key_iv: "eb5d9f73513d4ded9fb39334a2379490"
server_key_version: "encryptionkey@0"
I20260810 22:38:18.160543 8004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786401215402979-17317-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786401215402979-17317-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786401215402979-17317-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:38:18.176479 8004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:38:18.180290 8004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:38:18.180544 8004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:38:18.188318 8004 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45999
I20260810 22:38:18.188340 8075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45999 every 8 connection(s)
I20260810 22:38:18.189405 8004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
I20260810 22:38:18.192447 8076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:38:18.198304 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 8004
I20260810 22:38:18.198472 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786401215402979-17317-0/minicluster-data/master-0/wal/instance
I20260810 22:38:18.198705 8076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5476f5718dfb446ebb553d471b180bb9: Bootstrap starting.
I20260810 22:38:18.198799 17317 external_mini_cluster.cc:1428] Setting key 39af3058f2b8e808543560212895dd58
I20260810 22:38:18.201654 8076 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5476f5718dfb446ebb553d471b180bb9: Neither blocks nor log segments found. Creating new log.
I20260810 22:38:18.202495 8076 log.cc:824] T 00000000000000000000000000000000 P 5476f5718dfb446ebb553d471b180bb9: Log is configured to *not* fsync() on all Append() calls
I20260810 22:38:18.204658 8076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5476f5718dfb446ebb553d471b180bb9: No bootstrap required, opened a new log
Aug 10 22:38:18 dist-test-slave-77v9 krb5kdc[7990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401497, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 22:38:18.209213 8076 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5476f5718dfb446ebb553d471b180bb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5476f5718dfb446ebb553d471b180bb9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45999 } }
I20260810 22:38:18.209528 8076 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5476f5718dfb446ebb553d471b180bb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:38:18.209595 8076 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5476f5718dfb446ebb553d471b180bb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5476f5718dfb446ebb553d471b180bb9, State: Initialized, Role: FOLLOWER
I20260810 22:38:18.210106 8076 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5476f5718dfb446ebb553d471b180bb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5476f5718dfb446ebb553d471b180bb9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45999 } }
I20260810 22:38:18.210296 8076 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5476f5718dfb446ebb553d471b180bb9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:38:18.210391 8076 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5476f5718dfb446ebb553d471b180bb9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:38:18.210542 8076 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5476f5718dfb446ebb553d471b180bb9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:38:18.211565 8076 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5476f5718dfb446ebb553d471b180bb9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5476f5718dfb446ebb553d471b180bb9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45999 } }
I20260810 22:38:18.211943 8076 leader_election.cc:302] T 00000000000000000000000000000000 P 5476f5718dfb446ebb553d471b180bb9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5476f5718dfb446ebb553d471b180bb9; no voters:
I20260810 22:38:18.212289 8076 leader_election.cc:288] T 00000000000000000000000000000000 P 5476f5718dfb446ebb553d471b180bb9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:38:18.212425 8082 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5476f5718dfb446ebb553d471b180bb9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:38:18.212653 8082 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5476f5718dfb446ebb553d471b180bb9 [term 1 LEADER]: Becoming Leader. State: Replica: 5476f5718dfb446ebb553d471b180bb9, State: Running, Role: LEADER
I20260810 22:38:18.213724 8082 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5476f5718dfb446ebb553d471b180bb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5476f5718dfb446ebb553d471b180bb9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45999 } }
I20260810 22:38:18.213796 8076 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5476f5718dfb446ebb553d471b180bb9 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:38:18.214718 8079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:18.200351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50477 (local address 127.0.0.1:45999)
0810 22:38:18.200939 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:18.200956 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:18.200997 (+ 41us) server_negotiation.cc:407] Connection header received
0810 22:38:18.201689 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:18.201706 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:18.202127 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:18.202490 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:18.203417 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:18.204427 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:18.205426 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:18.205920 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:18.206118 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:18.209084 (+ 2966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:18.209090 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:18.209106 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:38:18.209138 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:18.211207 (+ 2069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:18.211693 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:18.211701 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:18.211709 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:18.211844 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:18.212168 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:18.212173 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:18.212175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:18.212790 (+ 615us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:18.213025 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:18.213329 (+ 304us) server_negotiation.cc:299] Negotiation successful
0810 22:38:18.213679 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":158,"threads_started":1}
I20260810 22:38:18.215719 8083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5476f5718dfb446ebb553d471b180bb9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5476f5718dfb446ebb553d471b180bb9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5476f5718dfb446ebb553d471b180bb9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45999 } } }
I20260810 22:38:18.215866 8083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5476f5718dfb446ebb553d471b180bb9 [sys.catalog]: This master's current role is: LEADER
I20260810 22:38:18.216181 8084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5476f5718dfb446ebb553d471b180bb9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5476f5718dfb446ebb553d471b180bb9. Latest consensus state: current_term: 1 leader_uuid: "5476f5718dfb446ebb553d471b180bb9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5476f5718dfb446ebb553d471b180bb9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45999 } } }
I20260810 22:38:18.217190 8084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5476f5718dfb446ebb553d471b180bb9 [sys.catalog]: This master's current role is: LEADER
I20260810 22:38:18.218904 17317 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260810 22:38:18.219398 8097 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5476f5718dfb446ebb553d471b180bb9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 22:38:18.219489 8097 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 22:38:18.220911 8096 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:38:18.221922 8096 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:38:18.228093 8096 catalog_manager.cc:1347] Generated new cluster ID: 2f7c86599f3d47b0adc642b6ca49a557
I20260810 22:38:18.228173 8096 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:38:18.232021 8079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:18.223618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52097 (local address 127.0.0.1:45999)
0810 22:38:18.223755 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:18.223759 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:18.223772 (+ 13us) server_negotiation.cc:407] Connection header received
0810 22:38:18.223808 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:18.223810 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:18.223848 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:18.223966 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:18.224723 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:18.225313 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:18.226301 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:18.226594 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:18.226705 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:18.228475 (+ 1770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:18.228478 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:18.228481 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:38:18.228508 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:18.230264 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:18.230775 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:18.230778 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:18.230780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:18.230862 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:18.231154 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:18.231157 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:18.231159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:18.231350 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:18.231475 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:18.231736 (+ 261us) server_negotiation.cc:299] Negotiation successful
0810 22:38:18.231857 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 22:38:18.252352 8096 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:38:18.253803 8096 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:38:18.259263 8096 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5476f5718dfb446ebb553d471b180bb9: Generated new TSK 0
I20260810 22:38:18.260172 8096 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:38:18.271566 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 8004
2026-08-10T22:38:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1576 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786401215402979-17317-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:38:18 dist-test-slave-77v9 krb5kdc[8109](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:38:18 dist-test-slave-77v9 krb5kdc[8109](info): set up 2 sockets
Aug 10 22:38:18 dist-test-slave-77v9 krb5kdc[8109](info): commencing operation
krb5kdc: starting...
W20260810 22:38:19.425148 17317 mini_kdc.cc:121] Time spent starting KDC: real 1.128s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Aug 10 22:38:19 dist-test-slave-77v9 krb5kdc[8109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401499, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:38:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:38:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786401215402979-17317-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786401215402979-17317-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 22:38:19.576221 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786401215402979-17317-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786401215402979-17317-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786401215402979-17317-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43749
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786401215402979-17317-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43749
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:38:19.695299 8127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:38:19.695837 8127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:38:19.695945 8127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:38:19.700685 8127 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:38:19.700776 8127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:38:19.700796 8127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:38:19.700878 8127 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:38:19.700909 8127 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:38:19.706755 8127 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786401215402979-17317-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786401215402979-17317-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43749
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786401215402979-17317-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786401215402979-17317-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:38:19.707981 8127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:38:19.709153 8127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:38:19.718346 8138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:38:19.718562 8127 server_base.cc:1034] running on GCE node
W20260810 22:38:19.718393 8134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:19.718358 8135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:38:19.719324 8127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:38:19.720536 8127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:38:19.721743 8127 hybrid_clock.cc:648] HybridClock initialized: now 1786401499721725 us; error 55 us; skew 500 ppm
Aug 10 22:38:19 dist-test-slave-77v9 krb5kdc[8109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401499, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:38:19.724906 8127 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 22:38:19.725982 8127 webserver.cc:460] Webserver started at http://127.0.0.1:46803/ using document root <none> and password file <none>
I20260810 22:38:19.726599 8127 fs_manager.cc:359] Metadata directory not provided
I20260810 22:38:19.726698 8127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:38:19.726989 8127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:38:19.729223 8127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786401215402979-17317-0/minicluster-data/master-0/data/instance:
uuid: "3c78acce3326427092e09f0dae8da16c"
format_stamp: "Formatted at 2026-08-10 22:38:19 on dist-test-slave-77v9"
server_key: "a108f86f48422f73482b23f4b11ef381"
server_key_iv: "b63fad9e0b5178eac91533636bb8b9f5"
server_key_version: "encryptionkey@0"
I20260810 22:38:19.729846 8127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786401215402979-17317-0/minicluster-data/master-0/wal/instance:
uuid: "3c78acce3326427092e09f0dae8da16c"
format_stamp: "Formatted at 2026-08-10 22:38:19 on dist-test-slave-77v9"
server_key: "a108f86f48422f73482b23f4b11ef381"
server_key_iv: "b63fad9e0b5178eac91533636bb8b9f5"
server_key_version: "encryptionkey@0"
I20260810 22:38:19.734059 8127 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 22:38:19.736898 8145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:19.738399 8127 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 22:38:19.738591 8127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786401215402979-17317-0/minicluster-data/master-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786401215402979-17317-0/minicluster-data/master-0/wal
uuid: "3c78acce3326427092e09f0dae8da16c"
format_stamp: "Formatted at 2026-08-10 22:38:19 on dist-test-slave-77v9"
server_key: "a108f86f48422f73482b23f4b11ef381"
server_key_iv: "b63fad9e0b5178eac91533636bb8b9f5"
server_key_version: "encryptionkey@0"
I20260810 22:38:19.738734 8127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786401215402979-17317-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786401215402979-17317-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786401215402979-17317-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:38:19.756695 8127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:38:19.761092 8127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:38:19.761384 8127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:38:19.771484 8127 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43749
I20260810 22:38:19.771802 8198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43749 every 8 connection(s)
I20260810 22:38:19.772701 8127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
I20260810 22:38:19.775061 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 8127
I20260810 22:38:19.775264 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786401215402979-17317-0/minicluster-data/master-0/wal/instance
I20260810 22:38:19.775599 17317 external_mini_cluster.cc:1428] Setting key 8b22d24562680559620109de9b34d9ab
I20260810 22:38:19.776024 8199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:38:19.784345 8199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c78acce3326427092e09f0dae8da16c: Bootstrap starting.
I20260810 22:38:19.786315 8202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:19.777286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43655 (local address 127.0.0.1:43749)
0810 22:38:19.777798 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:19.777817 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:19.777858 (+ 41us) server_negotiation.cc:407] Connection header received
0810 22:38:19.778621 (+ 763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:19.778642 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:19.779064 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:19.779411 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:19.779793 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:19.781080 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:19.782421 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:19.783018 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:19.783163 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:19.784226 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:19.784232 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:19.784247 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:38:19.784761 (+ 514us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:19.784894 (+ 133us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:38:19.784927 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:19.785015 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:19.785196 (+ 181us) server_negotiation.cc:299] Negotiation successful
0810 22:38:19.785454 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":178,"threads_started":1}
I20260810 22:38:19.787231 8199 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c78acce3326427092e09f0dae8da16c: Neither blocks nor log segments found. Creating new log.
I20260810 22:38:19.788100 8199 log.cc:824] T 00000000000000000000000000000000 P 3c78acce3326427092e09f0dae8da16c: Log is configured to *not* fsync() on all Append() calls
I20260810 22:38:19.789889 17317 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 22:38:19.790645 8199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c78acce3326427092e09f0dae8da16c: No bootstrap required, opened a new log
I20260810 22:38:19.795858 8199 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c78acce3326427092e09f0dae8da16c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c78acce3326427092e09f0dae8da16c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43749 } }
I20260810 22:38:19.796271 8199 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c78acce3326427092e09f0dae8da16c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:38:19.796350 8199 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c78acce3326427092e09f0dae8da16c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c78acce3326427092e09f0dae8da16c, State: Initialized, Role: FOLLOWER
I20260810 22:38:19.796906 8202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:19.792164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41267 (local address 127.0.0.1:43749)
0810 22:38:19.792297 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:19.792301 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:19.792551 (+ 250us) server_negotiation.cc:407] Connection header received
0810 22:38:19.792679 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:19.792681 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:19.792727 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:19.792803 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:19.793162 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:19.793724 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:19.794988 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:19.795228 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:19.795311 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:19.796327 (+ 1016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:19.796330 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:19.796348 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:38:19.796367 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:19.796426 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:38:19.796445 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:19.796515 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:19.796652 (+ 137us) server_negotiation.cc:299] Negotiation successful
0810 22:38:19.796721 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 22:38:19.796960 8199 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c78acce3326427092e09f0dae8da16c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c78acce3326427092e09f0dae8da16c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43749 } }
I20260810 22:38:19.797158 8199 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c78acce3326427092e09f0dae8da16c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:38:19.797227 8199 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c78acce3326427092e09f0dae8da16c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:38:19.797344 8199 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c78acce3326427092e09f0dae8da16c [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:38:19.798727 8199 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c78acce3326427092e09f0dae8da16c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c78acce3326427092e09f0dae8da16c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43749 } }
I20260810 22:38:19.799177 8199 leader_election.cc:302] T 00000000000000000000000000000000 P 3c78acce3326427092e09f0dae8da16c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c78acce3326427092e09f0dae8da16c; no voters:
I20260810 22:38:19.799499 8199 leader_election.cc:288] T 00000000000000000000000000000000 P 3c78acce3326427092e09f0dae8da16c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:38:19.799680 8209 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c78acce3326427092e09f0dae8da16c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:38:19.799933 8209 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c78acce3326427092e09f0dae8da16c [term 1 LEADER]: Becoming Leader. State: Replica: 3c78acce3326427092e09f0dae8da16c, State: Running, Role: LEADER
I20260810 22:38:19.800307 8209 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c78acce3326427092e09f0dae8da16c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c78acce3326427092e09f0dae8da16c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43749 } }
I20260810 22:38:19.800917 8199 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c78acce3326427092e09f0dae8da16c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:38:19.802253 8211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c78acce3326427092e09f0dae8da16c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c78acce3326427092e09f0dae8da16c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c78acce3326427092e09f0dae8da16c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43749 } } }
I20260810 22:38:19.802397 8211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c78acce3326427092e09f0dae8da16c [sys.catalog]: This master's current role is: LEADER
I20260810 22:38:19.802699 8212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c78acce3326427092e09f0dae8da16c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c78acce3326427092e09f0dae8da16c. Latest consensus state: current_term: 1 leader_uuid: "3c78acce3326427092e09f0dae8da16c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c78acce3326427092e09f0dae8da16c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43749 } } }
I20260810 22:38:19.802801 8212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c78acce3326427092e09f0dae8da16c [sys.catalog]: This master's current role is: LEADER
I20260810 22:38:19.802899 8219 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:38:19.806326 8219 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:38:19.812587 8219 catalog_manager.cc:1347] Generated new cluster ID: f9c489aa6afb4418bcd5314b82a21a5f
I20260810 22:38:19.812688 8219 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:38:19.820716 8219 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:38:19.821789 8219 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:38:19.827765 8219 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c78acce3326427092e09f0dae8da16c: Generated new TSK 0
I20260810 22:38:19.828572 8219 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:38:19.835449 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 8127
2026-08-10T22:38:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1563 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786401215402979-17317-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:38:19 dist-test-slave-77v9 krb5kdc[8231](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:38:19 dist-test-slave-77v9 krb5kdc[8231](info): set up 2 sockets
Aug 10 22:38:19 dist-test-slave-77v9 krb5kdc[8231](info): commencing operation
krb5kdc: starting...
W20260810 22:38:20.988876 17317 mini_kdc.cc:121] Time spent starting KDC: real 1.128s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Aug 10 22:38:21 dist-test-slave-77v9 krb5kdc[8231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:38:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:38:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786401215402979-17317-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786401215402979-17317-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 22:38:21.137387 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786401215402979-17317-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786401215402979-17317-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786401215402979-17317-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43589
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786401215402979-17317-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43589
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:38:21.265039 8245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:38:21.265355 8245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:38:21.265403 8245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:38:21.269781 8245 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:38:21.269881 8245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:38:21.269901 8245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:38:21.269919 8245 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:38:21.269935 8245 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:38:21.275772 8245 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786401215402979-17317-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786401215402979-17317-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43589
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786401215402979-17317-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786401215402979-17317-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:38:21.277132 8245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:38:21.278250 8245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:38:21.287904 8251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:21.288028 8250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:21.288041 8253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:38:21.288589 8245 server_base.cc:1034] running on GCE node
I20260810 22:38:21.289274 8245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:38:21.290534 8245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:38:21.291776 8245 hybrid_clock.cc:648] HybridClock initialized: now 1786401501291759 us; error 33 us; skew 500 ppm
Aug 10 22:38:21 dist-test-slave-77v9 krb5kdc[8231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401501, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:38:21.301190 8245 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 22:38:21.302371 8245 webserver.cc:460] Webserver started at http://127.0.0.1:40835/ using document root <none> and password file <none>
I20260810 22:38:21.302976 8245 fs_manager.cc:359] Metadata directory not provided
I20260810 22:38:21.303046 8245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:38:21.303231 8245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:38:21.305591 8245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786401215402979-17317-0/minicluster-data/master-0/data/instance:
uuid: "b9614c68e53a4b2d8143495962847577"
format_stamp: "Formatted at 2026-08-10 22:38:21 on dist-test-slave-77v9"
server_key: "007c16d529ab26d40e9c573356cad265"
server_key_iv: "67e8ccbecd3d43faa9f0822ca2702cb2"
server_key_version: "encryptionkey@0"
I20260810 22:38:21.306216 8245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786401215402979-17317-0/minicluster-data/master-0/wal/instance:
uuid: "b9614c68e53a4b2d8143495962847577"
format_stamp: "Formatted at 2026-08-10 22:38:21 on dist-test-slave-77v9"
server_key: "007c16d529ab26d40e9c573356cad265"
server_key_iv: "67e8ccbecd3d43faa9f0822ca2702cb2"
server_key_version: "encryptionkey@0"
I20260810 22:38:21.310544 8245 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 22:38:21.313539 8262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:21.315160 8245 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 22:38:21.315335 8245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786401215402979-17317-0/minicluster-data/master-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786401215402979-17317-0/minicluster-data/master-0/wal
uuid: "b9614c68e53a4b2d8143495962847577"
format_stamp: "Formatted at 2026-08-10 22:38:21 on dist-test-slave-77v9"
server_key: "007c16d529ab26d40e9c573356cad265"
server_key_iv: "67e8ccbecd3d43faa9f0822ca2702cb2"
server_key_version: "encryptionkey@0"
I20260810 22:38:21.315474 8245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786401215402979-17317-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786401215402979-17317-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786401215402979-17317-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:38:21.320868 8245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:38:21.324550 8245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:38:21.324949 8245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:38:21.333482 8245 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43589
I20260810 22:38:21.333547 8318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43589 every 8 connection(s)
I20260810 22:38:21.334617 8245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
I20260810 22:38:21.336419 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 8245
I20260810 22:38:21.336577 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786401215402979-17317-0/minicluster-data/master-0/wal/instance
I20260810 22:38:21.336946 17317 external_mini_cluster.cc:1428] Setting key 2a563cff03810cfe24b67d197ce0f84f
I20260810 22:38:21.338385 8319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:38:21.343343 8322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:21.338627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60917 (local address 127.0.0.1:43589)
0810 22:38:21.339090 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:21.339112 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:21.339158 (+ 46us) server_negotiation.cc:407] Connection header received
0810 22:38:21.340087 (+ 929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:21.340111 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:21.340483 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:21.340920 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:21.341737 (+ 817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:21.341743 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:21.341758 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:38:21.341771 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:21.341848 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:38:21.341879 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:21.342041 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:21.342188 (+ 147us) server_negotiation.cc:299] Negotiation successful
0810 22:38:21.342352 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":123,"threads_started":1}
I20260810 22:38:21.346076 8319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9614c68e53a4b2d8143495962847577: Bootstrap starting.
I20260810 22:38:21.347126 17317 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 22:38:21.349215 8319 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9614c68e53a4b2d8143495962847577: Neither blocks nor log segments found. Creating new log.
I20260810 22:38:21.350185 8319 log.cc:824] T 00000000000000000000000000000000 P b9614c68e53a4b2d8143495962847577: Log is configured to *not* fsync() on all Append() calls
I20260810 22:38:21.351774 8322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:21.349775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45779 (local address 127.0.0.1:43589)
0810 22:38:21.349936 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:21.349941 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:21.350156 (+ 215us) server_negotiation.cc:407] Connection header received
0810 22:38:21.350286 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:21.350288 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:21.350367 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:21.350467 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:21.351306 (+ 839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:21.351309 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:21.351311 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:38:21.351322 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:21.351346 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:38:21.351356 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:21.351450 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:21.351576 (+ 126us) server_negotiation.cc:299] Negotiation successful
0810 22:38:21.351610 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 22:38:21.353142 8319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9614c68e53a4b2d8143495962847577: No bootstrap required, opened a new log
I20260810 22:38:21.357090 8319 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9614c68e53a4b2d8143495962847577 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9614c68e53a4b2d8143495962847577" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43589 } }
I20260810 22:38:21.357405 8319 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9614c68e53a4b2d8143495962847577 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:38:21.357493 8319 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9614c68e53a4b2d8143495962847577 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9614c68e53a4b2d8143495962847577, State: Initialized, Role: FOLLOWER
I20260810 22:38:21.358037 8319 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9614c68e53a4b2d8143495962847577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9614c68e53a4b2d8143495962847577" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43589 } }
I20260810 22:38:21.358232 8319 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9614c68e53a4b2d8143495962847577 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:38:21.358307 8319 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9614c68e53a4b2d8143495962847577 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:38:21.358420 8319 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9614c68e53a4b2d8143495962847577 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:38:21.359531 8319 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9614c68e53a4b2d8143495962847577 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9614c68e53a4b2d8143495962847577" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43589 } }
I20260810 22:38:21.359925 8319 leader_election.cc:302] T 00000000000000000000000000000000 P b9614c68e53a4b2d8143495962847577 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9614c68e53a4b2d8143495962847577; no voters:
I20260810 22:38:21.360312 8319 leader_election.cc:288] T 00000000000000000000000000000000 P b9614c68e53a4b2d8143495962847577 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:38:21.360464 8330 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9614c68e53a4b2d8143495962847577 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:38:21.360754 8330 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9614c68e53a4b2d8143495962847577 [term 1 LEADER]: Becoming Leader. State: Replica: b9614c68e53a4b2d8143495962847577, State: Running, Role: LEADER
I20260810 22:38:21.361182 8330 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9614c68e53a4b2d8143495962847577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9614c68e53a4b2d8143495962847577" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43589 } }
I20260810 22:38:21.361778 8319 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9614c68e53a4b2d8143495962847577 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:38:21.363117 8332 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9614c68e53a4b2d8143495962847577 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9614c68e53a4b2d8143495962847577. Latest consensus state: current_term: 1 leader_uuid: "b9614c68e53a4b2d8143495962847577" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9614c68e53a4b2d8143495962847577" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43589 } } }
I20260810 22:38:21.363117 8331 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9614c68e53a4b2d8143495962847577 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9614c68e53a4b2d8143495962847577" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9614c68e53a4b2d8143495962847577" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43589 } } }
I20260810 22:38:21.363297 8332 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9614c68e53a4b2d8143495962847577 [sys.catalog]: This master's current role is: LEADER
I20260810 22:38:21.363315 8331 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9614c68e53a4b2d8143495962847577 [sys.catalog]: This master's current role is: LEADER
W20260810 22:38:21.367017 8345 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b9614c68e53a4b2d8143495962847577: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 22:38:21.367131 8345 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 22:38:21.367215 8340 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:38:21.367972 8340 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:38:21.373663 8340 catalog_manager.cc:1347] Generated new cluster ID: 964865ec86454c53add13e69a6346605
I20260810 22:38:21.373752 8340 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:38:21.381979 8340 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:38:21.382365 8340 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:38:21.390163 8340 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9614c68e53a4b2d8143495962847577: Generated new TSK 0
I20260810 22:38:21.391069 8340 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:38:21.404580 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 8245
2026-08-10T22:38:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1567 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786401215402979-17317-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:38:21 dist-test-slave-77v9 krb5kdc[8351](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:38:21 dist-test-slave-77v9 krb5kdc[8351](info): set up 2 sockets
Aug 10 22:38:21 dist-test-slave-77v9 krb5kdc[8351](info): commencing operation
krb5kdc: starting...
W20260810 22:38:22.531786 17317 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.002s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Aug 10 22:38:22 dist-test-slave-77v9 krb5kdc[8351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:38:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:38:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786401215402979-17317-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786401215402979-17317-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 22:38:22.681500 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786401215402979-17317-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786401215402979-17317-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786401215402979-17317-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46137
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786401215402979-17317-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46137
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:38:22.809448 8365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:38:22.809825 8365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:38:22.809938 8365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:38:22.814213 8365 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:38:22.814330 8365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:38:22.814373 8365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:38:22.814448 8365 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:38:22.814492 8365 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:38:22.820199 8365 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786401215402979-17317-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786401215402979-17317-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46137
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786401215402979-17317-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786401215402979-17317-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:38:22.821537 8365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:38:22.822656 8365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:38:22.831933 8374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:22.831954 8372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:22.831961 8370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:38:22.832290 8365 server_base.cc:1034] running on GCE node
I20260810 22:38:22.833056 8365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:38:22.834340 8365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:38:22.835573 8365 hybrid_clock.cc:648] HybridClock initialized: now 1786401502835504 us; error 86 us; skew 500 ppm
Aug 10 22:38:22 dist-test-slave-77v9 krb5kdc[8351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401502, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:38:22.839216 8365 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 22:38:22.840360 8365 webserver.cc:460] Webserver started at http://127.0.0.1:37565/ using document root <none> and password file <none>
I20260810 22:38:22.841200 8365 fs_manager.cc:359] Metadata directory not provided
I20260810 22:38:22.841315 8365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:38:22.841595 8365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:38:22.843956 8365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786401215402979-17317-0/minicluster-data/master-0/data/instance:
uuid: "47e76eb54e1648a68ab5cb99d986856f"
format_stamp: "Formatted at 2026-08-10 22:38:22 on dist-test-slave-77v9"
server_key: "40fb43b70615b42da6474ab5a168fecf"
server_key_iv: "5ad58a19aafcf2e9590949983a204bde"
server_key_version: "encryptionkey@0"
I20260810 22:38:22.844633 8365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786401215402979-17317-0/minicluster-data/master-0/wal/instance:
uuid: "47e76eb54e1648a68ab5cb99d986856f"
format_stamp: "Formatted at 2026-08-10 22:38:22 on dist-test-slave-77v9"
server_key: "40fb43b70615b42da6474ab5a168fecf"
server_key_iv: "5ad58a19aafcf2e9590949983a204bde"
server_key_version: "encryptionkey@0"
I20260810 22:38:22.849232 8365 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 22:38:22.852387 8381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:22.853960 8365 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:38:22.854144 8365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786401215402979-17317-0/minicluster-data/master-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786401215402979-17317-0/minicluster-data/master-0/wal
uuid: "47e76eb54e1648a68ab5cb99d986856f"
format_stamp: "Formatted at 2026-08-10 22:38:22 on dist-test-slave-77v9"
server_key: "40fb43b70615b42da6474ab5a168fecf"
server_key_iv: "5ad58a19aafcf2e9590949983a204bde"
server_key_version: "encryptionkey@0"
I20260810 22:38:22.854288 8365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786401215402979-17317-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786401215402979-17317-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786401215402979-17317-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:38:22.882287 8365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:38:22.885934 8365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:38:22.886166 8365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:38:22.894196 8365 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46137
I20260810 22:38:22.894228 8440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46137 every 8 connection(s)
I20260810 22:38:22.895239 8365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
I20260810 22:38:22.898147 8441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:38:22.900607 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 8365
I20260810 22:38:22.900769 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786401215402979-17317-0/minicluster-data/master-0/wal/instance
I20260810 22:38:22.901108 17317 external_mini_cluster.cc:1428] Setting key 6ad1699d2c3f9e078c6d609f8b42d4e5
I20260810 22:38:22.904345 8441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47e76eb54e1648a68ab5cb99d986856f: Bootstrap starting.
I20260810 22:38:22.906747 8441 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47e76eb54e1648a68ab5cb99d986856f: Neither blocks nor log segments found. Creating new log.
I20260810 22:38:22.907552 8441 log.cc:824] T 00000000000000000000000000000000 P 47e76eb54e1648a68ab5cb99d986856f: Log is configured to *not* fsync() on all Append() calls
I20260810 22:38:22.909801 8441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47e76eb54e1648a68ab5cb99d986856f: No bootstrap required, opened a new log
Aug 10 22:38:22 dist-test-slave-77v9 krb5kdc[8351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 22:38:22.913662 8441 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47e76eb54e1648a68ab5cb99d986856f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47e76eb54e1648a68ab5cb99d986856f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46137 } }
I20260810 22:38:22.913941 8441 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47e76eb54e1648a68ab5cb99d986856f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:38:22.914008 8441 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47e76eb54e1648a68ab5cb99d986856f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47e76eb54e1648a68ab5cb99d986856f, State: Initialized, Role: FOLLOWER
I20260810 22:38:22.914522 8441 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47e76eb54e1648a68ab5cb99d986856f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47e76eb54e1648a68ab5cb99d986856f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46137 } }
I20260810 22:38:22.914707 8441 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47e76eb54e1648a68ab5cb99d986856f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:38:22.914788 8441 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47e76eb54e1648a68ab5cb99d986856f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:38:22.914906 8441 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47e76eb54e1648a68ab5cb99d986856f [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:38:22.916030 8441 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47e76eb54e1648a68ab5cb99d986856f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47e76eb54e1648a68ab5cb99d986856f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46137 } }
I20260810 22:38:22.916409 8441 leader_election.cc:302] T 00000000000000000000000000000000 P 47e76eb54e1648a68ab5cb99d986856f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47e76eb54e1648a68ab5cb99d986856f; no voters:
I20260810 22:38:22.916752 8441 leader_election.cc:288] T 00000000000000000000000000000000 P 47e76eb54e1648a68ab5cb99d986856f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:38:22.916893 8446 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47e76eb54e1648a68ab5cb99d986856f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:38:22.917119 8446 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47e76eb54e1648a68ab5cb99d986856f [term 1 LEADER]: Becoming Leader. State: Replica: 47e76eb54e1648a68ab5cb99d986856f, State: Running, Role: LEADER
I20260810 22:38:22.917479 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:22.902675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.120:39481 (local address 127.0.0.1:46137)
0810 22:38:22.903159 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:22.903173 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:22.903209 (+ 36us) server_negotiation.cc:407] Connection header received
0810 22:38:22.903843 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:22.903859 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:22.904219 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:22.904574 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:22.905582 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:22.906620 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:22.907565 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:22.908168 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:22.908369 (+ 201us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:22.911385 (+ 3016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:22.911390 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:22.911403 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:38:22.911434 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:22.913765 (+ 2331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:22.914393 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:22.914399 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:22.914405 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:22.914531 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:22.914943 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:22.914946 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:22.914948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:22.915418 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:22.915629 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:22.915989 (+ 360us) server_negotiation.cc:299] Negotiation successful
0810 22:38:22.916250 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":143,"threads_started":1}
I20260810 22:38:22.917439 8446 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47e76eb54e1648a68ab5cb99d986856f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47e76eb54e1648a68ab5cb99d986856f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46137 } }
I20260810 22:38:22.918182 8441 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47e76eb54e1648a68ab5cb99d986856f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:38:22.919516 8447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47e76eb54e1648a68ab5cb99d986856f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47e76eb54e1648a68ab5cb99d986856f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47e76eb54e1648a68ab5cb99d986856f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46137 } } }
I20260810 22:38:22.919586 8448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47e76eb54e1648a68ab5cb99d986856f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47e76eb54e1648a68ab5cb99d986856f. Latest consensus state: current_term: 1 leader_uuid: "47e76eb54e1648a68ab5cb99d986856f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47e76eb54e1648a68ab5cb99d986856f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46137 } } }
I20260810 22:38:22.919643 8447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47e76eb54e1648a68ab5cb99d986856f [sys.catalog]: This master's current role is: LEADER
I20260810 22:38:22.919697 8448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47e76eb54e1648a68ab5cb99d986856f [sys.catalog]: This master's current role is: LEADER
I20260810 22:38:22.920176 8455 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:38:22.921263 17317 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 22:38:22.923702 8455 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:38:22.930907 8455 catalog_manager.cc:1347] Generated new cluster ID: b6fad2583ca344feb52a2ddc382fb567
I20260810 22:38:22.930994 8455 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:38:22.933202 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:22.923611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.120:58185 (local address 127.0.0.1:46137)
0810 22:38:22.923759 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:22.923764 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:22.924246 (+ 482us) server_negotiation.cc:407] Connection header received
0810 22:38:22.924372 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:22.924375 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:22.924431 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:22.924520 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:22.925487 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:22.926164 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:22.927032 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:22.927290 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:22.927409 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:22.929185 (+ 1776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:22.929194 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:22.929196 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:38:22.929222 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:22.930986 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:22.931630 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:22.931634 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:22.931636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:22.931736 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:22.932162 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:22.932165 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:22.932167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:22.932405 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:22.932593 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:22.932895 (+ 302us) server_negotiation.cc:299] Negotiation successful
0810 22:38:22.933013 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 22:38:22.943090 8455 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:38:22.944170 8455 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:38:22.954685 8455 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 47e76eb54e1648a68ab5cb99d986856f: Generated new TSK 0
I20260810 22:38:22.955928 8455 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:38:22.962623 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 8365
W20260810 22:38:22.972630 8349 connection.cc:535] client connection to 127.0.0.1:46137 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46137): Connection reset by peer (error 104)
2026-08-10T22:38:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1559 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786401215402979-17317-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:38:23 dist-test-slave-77v9 krb5kdc[8473](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:38:23 dist-test-slave-77v9 krb5kdc[8473](info): set up 2 sockets
Aug 10 22:38:23 dist-test-slave-77v9 krb5kdc[8473](info): commencing operation
krb5kdc: starting...
W20260810 22:38:24.122588 17317 mini_kdc.cc:121] Time spent starting KDC: real 1.135s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Aug 10 22:38:24 dist-test-slave-77v9 krb5kdc[8473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401504, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:38:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:38:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786401215402979-17317-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786401215402979-17317-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 22:38:24.265722 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786401215402979-17317-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786401215402979-17317-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786401215402979-17317-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45577
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786401215402979-17317-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45577
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:38:24.384615 8487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:38:24.384981 8487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:38:24.385072 8487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:38:24.389252 8487 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:38:24.389365 8487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:38:24.389403 8487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:38:24.389446 8487 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:38:24.389480 8487 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:38:24.395282 8487 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786401215402979-17317-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786401215402979-17317-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45577
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786401215402979-17317-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786401215402979-17317-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:38:24.396696 8487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:38:24.398025 8487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:38:24.407725 8494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:38:24.407773 8487 server_base.cc:1034] running on GCE node
W20260810 22:38:24.407711 8493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:24.407711 8496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:38:24.408670 8487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:38:24.409994 8487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:38:24.411230 8487 hybrid_clock.cc:648] HybridClock initialized: now 1786401504411206 us; error 35 us; skew 500 ppm
Aug 10 22:38:24 dist-test-slave-77v9 krb5kdc[8473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:38:24.419341 8487 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 22:38:24.420506 8487 webserver.cc:460] Webserver started at http://127.0.0.1:33683/ using document root <none> and password file <none>
I20260810 22:38:24.421234 8487 fs_manager.cc:359] Metadata directory not provided
I20260810 22:38:24.421343 8487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:38:24.421614 8487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:38:24.423923 8487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786401215402979-17317-0/minicluster-data/master-0/data/instance:
uuid: "d02bd3fc830c4a538ae4b054678522eb"
format_stamp: "Formatted at 2026-08-10 22:38:24 on dist-test-slave-77v9"
server_key: "c940477b7fe4ee2d841176946b0767ce"
server_key_iv: "609ca6e0a9903360ed536c32159642a2"
server_key_version: "encryptionkey@0"
I20260810 22:38:24.424576 8487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786401215402979-17317-0/minicluster-data/master-0/wal/instance:
uuid: "d02bd3fc830c4a538ae4b054678522eb"
format_stamp: "Formatted at 2026-08-10 22:38:24 on dist-test-slave-77v9"
server_key: "c940477b7fe4ee2d841176946b0767ce"
server_key_iv: "609ca6e0a9903360ed536c32159642a2"
server_key_version: "encryptionkey@0"
I20260810 22:38:24.429066 8487 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 22:38:24.431695 8504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:24.433131 8487 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:38:24.433321 8487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786401215402979-17317-0/minicluster-data/master-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786401215402979-17317-0/minicluster-data/master-0/wal
uuid: "d02bd3fc830c4a538ae4b054678522eb"
format_stamp: "Formatted at 2026-08-10 22:38:24 on dist-test-slave-77v9"
server_key: "c940477b7fe4ee2d841176946b0767ce"
server_key_iv: "609ca6e0a9903360ed536c32159642a2"
server_key_version: "encryptionkey@0"
I20260810 22:38:24.433462 8487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786401215402979-17317-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786401215402979-17317-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786401215402979-17317-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:38:24.459983 8487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:38:24.463641 8487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:38:24.463941 8487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:38:24.472616 8487 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45577
I20260810 22:38:24.472642 8562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45577 every 8 connection(s)
I20260810 22:38:24.473857 8487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
I20260810 22:38:24.474551 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 8487
I20260810 22:38:24.474725 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786401215402979-17317-0/minicluster-data/master-0/wal/instance
I20260810 22:38:24.475064 17317 external_mini_cluster.cc:1428] Setting key e36a6d5155cec407ae3b5cbe412d4de4
I20260810 22:38:24.477293 8563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:38:24.484426 8563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d02bd3fc830c4a538ae4b054678522eb: Bootstrap starting.
I20260810 22:38:24.485136 8566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:24.476629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.120:58503 (local address 127.0.0.1:45577)
0810 22:38:24.477091 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:24.477107 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:24.477142 (+ 35us) server_negotiation.cc:407] Connection header received
0810 22:38:24.477881 (+ 739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:24.477900 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:24.478243 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:24.478535 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:24.479019 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:24.480062 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:24.481384 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:24.481901 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:24.482020 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:24.483153 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:24.483158 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:24.483192 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:38:24.483595 (+ 403us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:24.483653 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:38:24.483676 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:24.483782 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:24.484045 (+ 263us) server_negotiation.cc:299] Negotiation successful
0810 22:38:24.484215 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":177,"threads_started":1}
I20260810 22:38:24.487259 8563 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d02bd3fc830c4a538ae4b054678522eb: Neither blocks nor log segments found. Creating new log.
I20260810 22:38:24.488205 8563 log.cc:824] T 00000000000000000000000000000000 P d02bd3fc830c4a538ae4b054678522eb: Log is configured to *not* fsync() on all Append() calls
I20260810 22:38:24.488725 17317 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 22:38:24.490672 8563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d02bd3fc830c4a538ae4b054678522eb: No bootstrap required, opened a new log
I20260810 22:38:24.495494 8566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:24.490955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.120:40063 (local address 127.0.0.1:45577)
0810 22:38:24.491101 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:24.491107 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:24.491353 (+ 246us) server_negotiation.cc:407] Connection header received
0810 22:38:24.491443 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:24.491447 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:24.491499 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:24.491589 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:24.491843 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:24.492624 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:24.493610 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:24.493983 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:24.494107 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:24.494997 (+ 890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:24.495001 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:24.495004 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:38:24.495020 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:24.495053 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:38:24.495069 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:24.495181 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:24.495292 (+ 111us) server_negotiation.cc:299] Negotiation successful
0810 22:38:24.495336 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 22:38:24.495573 8563 raft_consensus.cc:348] T 00000000000000000000000000000000 P d02bd3fc830c4a538ae4b054678522eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d02bd3fc830c4a538ae4b054678522eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45577 } }
I20260810 22:38:24.495919 8563 raft_consensus.cc:374] T 00000000000000000000000000000000 P d02bd3fc830c4a538ae4b054678522eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:38:24.495991 8563 raft_consensus.cc:729] T 00000000000000000000000000000000 P d02bd3fc830c4a538ae4b054678522eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d02bd3fc830c4a538ae4b054678522eb, State: Initialized, Role: FOLLOWER
I20260810 22:38:24.496450 8563 consensus_queue.cc:260] T 00000000000000000000000000000000 P d02bd3fc830c4a538ae4b054678522eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d02bd3fc830c4a538ae4b054678522eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45577 } }
I20260810 22:38:24.496620 8563 raft_consensus.cc:388] T 00000000000000000000000000000000 P d02bd3fc830c4a538ae4b054678522eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:38:24.496701 8563 raft_consensus.cc:482] T 00000000000000000000000000000000 P d02bd3fc830c4a538ae4b054678522eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:38:24.496835 8563 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d02bd3fc830c4a538ae4b054678522eb [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:38:24.497926 8563 raft_consensus.cc:504] T 00000000000000000000000000000000 P d02bd3fc830c4a538ae4b054678522eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d02bd3fc830c4a538ae4b054678522eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45577 } }
I20260810 22:38:24.498343 8563 leader_election.cc:302] T 00000000000000000000000000000000 P d02bd3fc830c4a538ae4b054678522eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d02bd3fc830c4a538ae4b054678522eb; no voters:
I20260810 22:38:24.498750 8563 leader_election.cc:288] T 00000000000000000000000000000000 P d02bd3fc830c4a538ae4b054678522eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:38:24.498867 8573 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d02bd3fc830c4a538ae4b054678522eb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:38:24.499146 8573 raft_consensus.cc:686] T 00000000000000000000000000000000 P d02bd3fc830c4a538ae4b054678522eb [term 1 LEADER]: Becoming Leader. State: Replica: d02bd3fc830c4a538ae4b054678522eb, State: Running, Role: LEADER
I20260810 22:38:24.499678 8573 consensus_queue.cc:237] T 00000000000000000000000000000000 P d02bd3fc830c4a538ae4b054678522eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d02bd3fc830c4a538ae4b054678522eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45577 } }
I20260810 22:38:24.500187 8563 sys_catalog.cc:564] T 00000000000000000000000000000000 P d02bd3fc830c4a538ae4b054678522eb [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:38:24.501658 8575 sys_catalog.cc:455] T 00000000000000000000000000000000 P d02bd3fc830c4a538ae4b054678522eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader d02bd3fc830c4a538ae4b054678522eb. Latest consensus state: current_term: 1 leader_uuid: "d02bd3fc830c4a538ae4b054678522eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d02bd3fc830c4a538ae4b054678522eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45577 } } }
I20260810 22:38:24.501719 8574 sys_catalog.cc:455] T 00000000000000000000000000000000 P d02bd3fc830c4a538ae4b054678522eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d02bd3fc830c4a538ae4b054678522eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d02bd3fc830c4a538ae4b054678522eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45577 } } }
I20260810 22:38:24.501806 8575 sys_catalog.cc:458] T 00000000000000000000000000000000 P d02bd3fc830c4a538ae4b054678522eb [sys.catalog]: This master's current role is: LEADER
I20260810 22:38:24.501833 8574 sys_catalog.cc:458] T 00000000000000000000000000000000 P d02bd3fc830c4a538ae4b054678522eb [sys.catalog]: This master's current role is: LEADER
I20260810 22:38:24.502224 8582 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:38:24.505440 8582 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:38:24.511978 8582 catalog_manager.cc:1347] Generated new cluster ID: 56678927daa94e918d60d8779db0ea82
I20260810 22:38:24.512082 8582 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:38:24.547374 8582 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:38:24.549057 8582 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:38:24.555124 8582 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d02bd3fc830c4a538ae4b054678522eb: Generated new TSK 0
I20260810 22:38:24.556196 8582 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:38:24.569324 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 8487
W20260810 22:38:24.581935 8471 connection.cc:535] client connection to 127.0.0.1:45577 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45577): Connection reset by peer (error 104)
2026-08-10T22:38:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1609 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786401215402979-17317-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:38:24 dist-test-slave-77v9 krb5kdc[8595](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:38:24 dist-test-slave-77v9 krb5kdc[8595](info): set up 2 sockets
Aug 10 22:38:24 dist-test-slave-77v9 krb5kdc[8595](info): commencing operation
krb5kdc: starting...
W20260810 22:38:25.744129 17317 mini_kdc.cc:121] Time spent starting KDC: real 1.147s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Aug 10 22:38:25 dist-test-slave-77v9 krb5kdc[8595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:38:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:38:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786401215402979-17317-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786401215402979-17317-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 22:38:25.889088 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786401215402979-17317-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786401215402979-17317-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786401215402979-17317-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43317
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786401215402979-17317-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43317
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:38:26.007829 8609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:38:26.008188 8609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:38:26.008276 8609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:38:26.012444 8609 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:38:26.012563 8609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:38:26.012601 8609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:38:26.012641 8609 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:38:26.012681 8609 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:38:26.019209 8609 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786401215402979-17317-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786401215402979-17317-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43317
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786401215402979-17317-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786401215402979-17317-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:38:26.020498 8609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:38:26.021647 8609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:38:26.031004 8614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:26.031015 8617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:26.031028 8615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:38:26.031216 8609 server_base.cc:1034] running on GCE node
I20260810 22:38:26.032138 8609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:38:26.033555 8609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:38:26.034795 8609 hybrid_clock.cc:648] HybridClock initialized: now 1786401506034768 us; error 47 us; skew 500 ppm
Aug 10 22:38:26 dist-test-slave-77v9 krb5kdc[8595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401506, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:38:26.038340 8609 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 22:38:26.039506 8609 webserver.cc:460] Webserver started at http://127.0.0.1:41815/ using document root <none> and password file <none>
I20260810 22:38:26.040159 8609 fs_manager.cc:359] Metadata directory not provided
I20260810 22:38:26.040270 8609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:38:26.040539 8609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:38:26.043001 8609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786401215402979-17317-0/minicluster-data/master-0/data/instance:
uuid: "6a07067f43814de6837a00fcbd1e4381"
format_stamp: "Formatted at 2026-08-10 22:38:26 on dist-test-slave-77v9"
server_key: "4d0ac45bb455305557ef8c4f489d4f1f"
server_key_iv: "a44ef5ab38a020ca8a3fa709d12c7aa1"
server_key_version: "encryptionkey@0"
I20260810 22:38:26.043669 8609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786401215402979-17317-0/minicluster-data/master-0/wal/instance:
uuid: "6a07067f43814de6837a00fcbd1e4381"
format_stamp: "Formatted at 2026-08-10 22:38:26 on dist-test-slave-77v9"
server_key: "4d0ac45bb455305557ef8c4f489d4f1f"
server_key_iv: "a44ef5ab38a020ca8a3fa709d12c7aa1"
server_key_version: "encryptionkey@0"
I20260810 22:38:26.048172 8609 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260810 22:38:26.051244 8624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:26.052700 8609 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 22:38:26.053174 8609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786401215402979-17317-0/minicluster-data/master-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786401215402979-17317-0/minicluster-data/master-0/wal
uuid: "6a07067f43814de6837a00fcbd1e4381"
format_stamp: "Formatted at 2026-08-10 22:38:26 on dist-test-slave-77v9"
server_key: "4d0ac45bb455305557ef8c4f489d4f1f"
server_key_iv: "a44ef5ab38a020ca8a3fa709d12c7aa1"
server_key_version: "encryptionkey@0"
I20260810 22:38:26.053326 8609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786401215402979-17317-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786401215402979-17317-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786401215402979-17317-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:38:26.058838 8609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:38:26.062567 8609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:38:26.062800 8609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:38:26.071107 8609 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43317
I20260810 22:38:26.071158 8681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43317 every 8 connection(s)
I20260810 22:38:26.072192 8609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
I20260810 22:38:26.075176 8682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:38:26.077884 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 8609
I20260810 22:38:26.078058 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786401215402979-17317-0/minicluster-data/master-0/wal/instance
I20260810 22:38:26.078357 17317 external_mini_cluster.cc:1428] Setting key 6720ee719e7f1a7f7dc5a66562b76535
I20260810 22:38:26.081964 8682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a07067f43814de6837a00fcbd1e4381: Bootstrap starting.
I20260810 22:38:26.084420 8687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:26.079937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.120:49449 (local address 127.0.0.1:43317)
0810 22:38:26.080442 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:26.080463 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:26.080511 (+ 48us) server_negotiation.cc:407] Connection header received
0810 22:38:26.081403 (+ 892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:26.081427 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:26.081801 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:26.082195 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:26.083005 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:26.083010 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:26.083024 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:38:26.083035 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:26.083099 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:38:26.083125 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:26.083270 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:26.083349 (+ 79us) server_negotiation.cc:299] Negotiation successful
0810 22:38:26.083512 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":155,"threads_started":1}
I20260810 22:38:26.084846 8682 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a07067f43814de6837a00fcbd1e4381: Neither blocks nor log segments found. Creating new log.
I20260810 22:38:26.085732 8682 log.cc:824] T 00000000000000000000000000000000 P 6a07067f43814de6837a00fcbd1e4381: Log is configured to *not* fsync() on all Append() calls
I20260810 22:38:26.087636 17317 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 22:38:26.087976 8682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a07067f43814de6837a00fcbd1e4381: No bootstrap required, opened a new log
I20260810 22:38:26.091627 8687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:26.089904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.120:38041 (local address 127.0.0.1:43317)
0810 22:38:26.090030 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:26.090034 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:26.090183 (+ 149us) server_negotiation.cc:407] Connection header received
0810 22:38:26.090316 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:26.090318 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:26.090347 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:26.090416 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:26.091171 (+ 755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:26.091174 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:26.091176 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:38:26.091186 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:26.091210 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:38:26.091221 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:26.091299 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:26.091428 (+ 129us) server_negotiation.cc:299] Negotiation successful
0810 22:38:26.091463 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 22:38:26.092526 8682 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a07067f43814de6837a00fcbd1e4381 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a07067f43814de6837a00fcbd1e4381" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43317 } }
I20260810 22:38:26.092877 8682 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a07067f43814de6837a00fcbd1e4381 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:38:26.092964 8682 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a07067f43814de6837a00fcbd1e4381 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a07067f43814de6837a00fcbd1e4381, State: Initialized, Role: FOLLOWER
I20260810 22:38:26.093380 8682 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a07067f43814de6837a00fcbd1e4381 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a07067f43814de6837a00fcbd1e4381" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43317 } }
I20260810 22:38:26.093533 8682 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a07067f43814de6837a00fcbd1e4381 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:38:26.093605 8682 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a07067f43814de6837a00fcbd1e4381 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:38:26.093704 8682 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a07067f43814de6837a00fcbd1e4381 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:38:26.094936 8682 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a07067f43814de6837a00fcbd1e4381 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a07067f43814de6837a00fcbd1e4381" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43317 } }
I20260810 22:38:26.095247 8682 leader_election.cc:302] T 00000000000000000000000000000000 P 6a07067f43814de6837a00fcbd1e4381 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a07067f43814de6837a00fcbd1e4381; no voters:
I20260810 22:38:26.095542 8682 leader_election.cc:288] T 00000000000000000000000000000000 P 6a07067f43814de6837a00fcbd1e4381 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:38:26.095708 8694 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a07067f43814de6837a00fcbd1e4381 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:38:26.096031 8694 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a07067f43814de6837a00fcbd1e4381 [term 1 LEADER]: Becoming Leader. State: Replica: 6a07067f43814de6837a00fcbd1e4381, State: Running, Role: LEADER
I20260810 22:38:26.096354 8694 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a07067f43814de6837a00fcbd1e4381 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a07067f43814de6837a00fcbd1e4381" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43317 } }
I20260810 22:38:26.096956 8682 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a07067f43814de6837a00fcbd1e4381 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:38:26.098299 8696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a07067f43814de6837a00fcbd1e4381 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a07067f43814de6837a00fcbd1e4381. Latest consensus state: current_term: 1 leader_uuid: "6a07067f43814de6837a00fcbd1e4381" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a07067f43814de6837a00fcbd1e4381" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43317 } } }
I20260810 22:38:26.098307 8695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a07067f43814de6837a00fcbd1e4381 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a07067f43814de6837a00fcbd1e4381" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a07067f43814de6837a00fcbd1e4381" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43317 } } }
I20260810 22:38:26.098523 8696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a07067f43814de6837a00fcbd1e4381 [sys.catalog]: This master's current role is: LEADER
I20260810 22:38:26.098531 8695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a07067f43814de6837a00fcbd1e4381 [sys.catalog]: This master's current role is: LEADER
W20260810 22:38:26.102095 8710 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6a07067f43814de6837a00fcbd1e4381: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 22:38:26.102166 8710 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 22:38:26.102253 8705 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:38:26.103009 8705 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:38:26.109560 8705 catalog_manager.cc:1347] Generated new cluster ID: 351940ba999f47f9b675598488732680
I20260810 22:38:26.109654 8705 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:38:26.145284 8705 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:38:26.145771 8705 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:38:26.152119 8705 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a07067f43814de6837a00fcbd1e4381: Generated new TSK 0
I20260810 22:38:26.152988 8705 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:38:26.159091 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 8609
2026-08-10T22:38:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1585 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786401215402979-17317-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:38:26 dist-test-slave-77v9 krb5kdc[8716](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:38:26 dist-test-slave-77v9 krb5kdc[8716](info): set up 2 sockets
Aug 10 22:38:26 dist-test-slave-77v9 krb5kdc[8716](info): commencing operation
krb5kdc: starting...
W20260810 22:38:27.299734 17317 mini_kdc.cc:121] Time spent starting KDC: real 1.117s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Aug 10 22:38:27 dist-test-slave-77v9 krb5kdc[8716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401507, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:38:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:38:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786401215402979-17317-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786401215402979-17317-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 22:38:27.445550 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786401215402979-17317-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786401215402979-17317-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786401215402979-17317-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41761
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786401215402979-17317-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41761
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:38:27.568678 8730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:38:27.569044 8730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:38:27.569139 8730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:38:27.573443 8730 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:38:27.573581 8730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:38:27.573622 8730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:38:27.573669 8730 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:38:27.573705 8730 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:38:27.579561 8730 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786401215402979-17317-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786401215402979-17317-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41761
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786401215402979-17317-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786401215402979-17317-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:38:27.580901 8730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:38:27.582022 8730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 22:38:27.592041 8730 server_base.cc:1034] running on GCE node
W20260810 22:38:27.592031 8735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:27.592247 8738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:27.592031 8736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:38:27.593001 8730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:38:27.594434 8730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:38:27.595656 8730 hybrid_clock.cc:648] HybridClock initialized: now 1786401507595642 us; error 31 us; skew 500 ppm
Aug 10 22:38:27 dist-test-slave-77v9 krb5kdc[8716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401507, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:38:27.604460 8730 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 22:38:27.605682 8730 webserver.cc:460] Webserver started at http://127.0.0.1:45877/ using document root <none> and password file <none>
I20260810 22:38:27.606365 8730 fs_manager.cc:359] Metadata directory not provided
I20260810 22:38:27.606496 8730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:38:27.606773 8730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:38:27.609177 8730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786401215402979-17317-0/minicluster-data/master-0/data/instance:
uuid: "fe6c5417121a4e448a9cdb4962e3b16a"
format_stamp: "Formatted at 2026-08-10 22:38:27 on dist-test-slave-77v9"
server_key: "5cb0396312e9f0626c03554da8e64fb7"
server_key_iv: "e832d0c8fa120523774899d1c370ff5e"
server_key_version: "encryptionkey@0"
I20260810 22:38:27.609815 8730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786401215402979-17317-0/minicluster-data/master-0/wal/instance:
uuid: "fe6c5417121a4e448a9cdb4962e3b16a"
format_stamp: "Formatted at 2026-08-10 22:38:27 on dist-test-slave-77v9"
server_key: "5cb0396312e9f0626c03554da8e64fb7"
server_key_iv: "e832d0c8fa120523774899d1c370ff5e"
server_key_version: "encryptionkey@0"
I20260810 22:38:27.614004 8730 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 22:38:27.616626 8746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:27.618172 8730 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 22:38:27.618358 8730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786401215402979-17317-0/minicluster-data/master-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786401215402979-17317-0/minicluster-data/master-0/wal
uuid: "fe6c5417121a4e448a9cdb4962e3b16a"
format_stamp: "Formatted at 2026-08-10 22:38:27 on dist-test-slave-77v9"
server_key: "5cb0396312e9f0626c03554da8e64fb7"
server_key_iv: "e832d0c8fa120523774899d1c370ff5e"
server_key_version: "encryptionkey@0"
I20260810 22:38:27.618505 8730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786401215402979-17317-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786401215402979-17317-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786401215402979-17317-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:38:27.632154 8730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:38:27.635504 8730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:38:27.635739 8730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:38:27.644142 8730 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41761
I20260810 22:38:27.644209 8803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41761 every 8 connection(s)
I20260810 22:38:27.645264 8730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
I20260810 22:38:27.648465 8804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:38:27.654556 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 8730
I20260810 22:38:27.654726 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786401215402979-17317-0/minicluster-data/master-0/wal/instance
I20260810 22:38:27.654726 8804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe6c5417121a4e448a9cdb4962e3b16a: Bootstrap starting.
I20260810 22:38:27.655068 17317 external_mini_cluster.cc:1428] Setting key 769a134938c3da4846297f6782cc659d
I20260810 22:38:27.657912 8804 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe6c5417121a4e448a9cdb4962e3b16a: Neither blocks nor log segments found. Creating new log.
I20260810 22:38:27.658876 8804 log.cc:824] T 00000000000000000000000000000000 P fe6c5417121a4e448a9cdb4962e3b16a: Log is configured to *not* fsync() on all Append() calls
I20260810 22:38:27.661403 8804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe6c5417121a4e448a9cdb4962e3b16a: No bootstrap required, opened a new log
Aug 10 22:38:27 dist-test-slave-77v9 krb5kdc[8716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401507, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 22:38:27.665999 8804 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe6c5417121a4e448a9cdb4962e3b16a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe6c5417121a4e448a9cdb4962e3b16a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41761 } }
I20260810 22:38:27.666318 8804 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe6c5417121a4e448a9cdb4962e3b16a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:38:27.666409 8804 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe6c5417121a4e448a9cdb4962e3b16a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe6c5417121a4e448a9cdb4962e3b16a, State: Initialized, Role: FOLLOWER
I20260810 22:38:27.666986 8804 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe6c5417121a4e448a9cdb4962e3b16a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe6c5417121a4e448a9cdb4962e3b16a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41761 } }
I20260810 22:38:27.667167 8804 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe6c5417121a4e448a9cdb4962e3b16a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:38:27.667237 8804 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe6c5417121a4e448a9cdb4962e3b16a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:38:27.667346 8804 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe6c5417121a4e448a9cdb4962e3b16a [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:38:27.668741 8804 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe6c5417121a4e448a9cdb4962e3b16a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe6c5417121a4e448a9cdb4962e3b16a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41761 } }
I20260810 22:38:27.669214 8804 leader_election.cc:302] T 00000000000000000000000000000000 P fe6c5417121a4e448a9cdb4962e3b16a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe6c5417121a4e448a9cdb4962e3b16a; no voters:
I20260810 22:38:27.669610 8804 leader_election.cc:288] T 00000000000000000000000000000000 P fe6c5417121a4e448a9cdb4962e3b16a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:38:27.669808 8810 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe6c5417121a4e448a9cdb4962e3b16a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:38:27.670073 8810 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe6c5417121a4e448a9cdb4962e3b16a [term 1 LEADER]: Becoming Leader. State: Replica: fe6c5417121a4e448a9cdb4962e3b16a, State: Running, Role: LEADER
I20260810 22:38:27.670488 8810 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe6c5417121a4e448a9cdb4962e3b16a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe6c5417121a4e448a9cdb4962e3b16a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41761 } }
I20260810 22:38:27.671103 8804 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe6c5417121a4e448a9cdb4962e3b16a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:38:27.672210 8812 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe6c5417121a4e448a9cdb4962e3b16a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe6c5417121a4e448a9cdb4962e3b16a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe6c5417121a4e448a9cdb4962e3b16a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41761 } } }
I20260810 22:38:27.672436 8812 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe6c5417121a4e448a9cdb4962e3b16a [sys.catalog]: This master's current role is: LEADER
I20260810 22:38:27.672483 8813 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe6c5417121a4e448a9cdb4962e3b16a [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe6c5417121a4e448a9cdb4962e3b16a. Latest consensus state: current_term: 1 leader_uuid: "fe6c5417121a4e448a9cdb4962e3b16a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe6c5417121a4e448a9cdb4962e3b16a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41761 } } }
I20260810 22:38:27.672680 8813 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe6c5417121a4e448a9cdb4962e3b16a [sys.catalog]: This master's current role is: LEADER
I20260810 22:38:27.673048 8821 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:38:27.673444 8808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:27.656642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.120:37351 (local address 127.0.0.1:41761)
0810 22:38:27.657367 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:27.657387 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:27.657429 (+ 42us) server_negotiation.cc:407] Connection header received
0810 22:38:27.658500 (+ 1071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:27.658525 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:27.659002 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:27.659405 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:27.660411 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:27.661918 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:27.662880 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:27.663627 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:27.663815 (+ 188us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:27.667034 (+ 3219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:27.667041 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:27.667058 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:38:27.667097 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:27.669517 (+ 2420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:27.670243 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:27.670250 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:27.670256 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:27.670408 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:27.670830 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:27.670834 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:27.670836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:27.671417 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:27.671631 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:27.672035 (+ 404us) server_negotiation.cc:299] Negotiation successful
0810 22:38:27.672280 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":361,"threads_started":1}
I20260810 22:38:27.676966 8821 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:38:27.677016 17317 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 22:38:27.685092 8821 catalog_manager.cc:1347] Generated new cluster ID: 7f0161972fa742d6a4b4ac3feb7db98d
I20260810 22:38:27.685184 8821 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:38:27.689077 8808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:27.679477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.120:44687 (local address 127.0.0.1:41761)
0810 22:38:27.679592 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:27.679597 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:27.679892 (+ 295us) server_negotiation.cc:407] Connection header received
0810 22:38:27.680029 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:27.680035 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:27.680102 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:27.680194 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:27.681116 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:27.682402 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:27.683244 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:27.683606 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:27.683724 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:27.685577 (+ 1853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:27.685581 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:27.685584 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:38:27.685609 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:27.687161 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:27.687680 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:27.687690 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:27.687692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:27.687776 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:27.688116 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:27.688119 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:27.688121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:27.688305 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:27.688460 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:27.688749 (+ 289us) server_negotiation.cc:299] Negotiation successful
0810 22:38:27.688881 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 22:38:27.708891 8821 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:38:27.709993 8821 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:38:27.729251 8821 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe6c5417121a4e448a9cdb4962e3b16a: Generated new TSK 0
I20260810 22:38:27.730208 8821 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:38:27.744683 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 8730
W20260810 22:38:27.755021 8714 connection.cc:535] client connection to 127.0.0.1:41761 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41761): Connection reset by peer (error 104)
2026-08-10T22:38:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1587 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786401215402979-17317-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:38:27 dist-test-slave-77v9 krb5kdc[8839](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:38:27 dist-test-slave-77v9 krb5kdc[8839](info): set up 2 sockets
Aug 10 22:38:27 dist-test-slave-77v9 krb5kdc[8839](info): commencing operation
krb5kdc: starting...
W20260810 22:38:28.891482 17317 mini_kdc.cc:121] Time spent starting KDC: real 1.121s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Aug 10 22:38:28 dist-test-slave-77v9 krb5kdc[8839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:38:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:38:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786401215402979-17317-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786401215402979-17317-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 22:38:29.037161 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786401215402979-17317-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786401215402979-17317-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786401215402979-17317-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42071
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786401215402979-17317-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42071
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:38:29.155344 8852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:38:29.155711 8852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:38:29.155798 8852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:38:29.160058 8852 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:38:29.160180 8852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:38:29.160216 8852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:38:29.160259 8852 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:38:29.160291 8852 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:38:29.166038 8852 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786401215402979-17317-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786401215402979-17317-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42071
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786401215402979-17317-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786401215402979-17317-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:38:29.167287 8852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:38:29.168401 8852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:38:29.177330 8861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:29.177338 8858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:38:29.177466 8852 server_base.cc:1034] running on GCE node
W20260810 22:38:29.177346 8859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:38:29.178231 8852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:38:29.179509 8852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:38:29.180740 8852 hybrid_clock.cc:648] HybridClock initialized: now 1786401509180683 us; error 65 us; skew 500 ppm
Aug 10 22:38:29 dist-test-slave-77v9 krb5kdc[8839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401509, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:38:29.183905 8852 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 22:38:29.184959 8852 webserver.cc:460] Webserver started at http://127.0.0.1:39609/ using document root <none> and password file <none>
I20260810 22:38:29.185551 8852 fs_manager.cc:359] Metadata directory not provided
I20260810 22:38:29.185648 8852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:38:29.185889 8852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:38:29.188055 8852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786401215402979-17317-0/minicluster-data/master-0/data/instance:
uuid: "9987237255af46a8af7b955ba27d8d27"
format_stamp: "Formatted at 2026-08-10 22:38:29 on dist-test-slave-77v9"
server_key: "f9894d0b94869633e1c9507950c25c1d"
server_key_iv: "3308b2105fc40c1ac3267ecbe3af3307"
server_key_version: "encryptionkey@0"
I20260810 22:38:29.188664 8852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786401215402979-17317-0/minicluster-data/master-0/wal/instance:
uuid: "9987237255af46a8af7b955ba27d8d27"
format_stamp: "Formatted at 2026-08-10 22:38:29 on dist-test-slave-77v9"
server_key: "f9894d0b94869633e1c9507950c25c1d"
server_key_iv: "3308b2105fc40c1ac3267ecbe3af3307"
server_key_version: "encryptionkey@0"
I20260810 22:38:29.192665 8852 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.002s
I20260810 22:38:29.195257 8868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:29.196682 8852 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 22:38:29.196943 8852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786401215402979-17317-0/minicluster-data/master-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786401215402979-17317-0/minicluster-data/master-0/wal
uuid: "9987237255af46a8af7b955ba27d8d27"
format_stamp: "Formatted at 2026-08-10 22:38:29 on dist-test-slave-77v9"
server_key: "f9894d0b94869633e1c9507950c25c1d"
server_key_iv: "3308b2105fc40c1ac3267ecbe3af3307"
server_key_version: "encryptionkey@0"
I20260810 22:38:29.197077 8852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786401215402979-17317-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786401215402979-17317-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786401215402979-17317-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:38:29.207875 8852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:38:29.211525 8852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:38:29.211781 8852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:38:29.219599 8852 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42071
I20260810 22:38:29.219625 8927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42071 every 8 connection(s)
I20260810 22:38:29.220978 8852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
I20260810 22:38:29.224068 8928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:38:29.225627 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 8852
I20260810 22:38:29.225797 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786401215402979-17317-0/minicluster-data/master-0/wal/instance
I20260810 22:38:29.226104 17317 external_mini_cluster.cc:1428] Setting key d3a36721beacbc19cbe37a537ae87637
I20260810 22:38:29.231441 8928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9987237255af46a8af7b955ba27d8d27: Bootstrap starting.
I20260810 22:38:29.234388 8928 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9987237255af46a8af7b955ba27d8d27: Neither blocks nor log segments found. Creating new log.
I20260810 22:38:29.235332 8928 log.cc:824] T 00000000000000000000000000000000 P 9987237255af46a8af7b955ba27d8d27: Log is configured to *not* fsync() on all Append() calls
I20260810 22:38:29.237323 8932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:29.227759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.120:60555 (local address 127.0.0.1:42071)
0810 22:38:29.228325 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:29.228348 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:29.228395 (+ 47us) server_negotiation.cc:407] Connection header received
0810 22:38:29.229330 (+ 935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:29.229356 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:29.229822 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:29.230217 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:29.230661 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:29.232102 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:29.233193 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:29.233916 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:29.234093 (+ 177us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:29.235059 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:29.235065 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:29.235082 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:38:29.235630 (+ 548us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:29.235701 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:38:29.235738 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:29.235899 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:29.236128 (+ 229us) server_negotiation.cc:299] Negotiation successful
0810 22:38:29.236317 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":178,"threads_started":1}
I20260810 22:38:29.237919 8928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9987237255af46a8af7b955ba27d8d27: No bootstrap required, opened a new log
I20260810 22:38:29.240890 17317 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 22:38:29.242664 8928 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9987237255af46a8af7b955ba27d8d27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9987237255af46a8af7b955ba27d8d27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42071 } }
I20260810 22:38:29.242975 8928 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9987237255af46a8af7b955ba27d8d27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:38:29.243137 8928 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9987237255af46a8af7b955ba27d8d27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9987237255af46a8af7b955ba27d8d27, State: Initialized, Role: FOLLOWER
I20260810 22:38:29.243680 8928 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9987237255af46a8af7b955ba27d8d27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9987237255af46a8af7b955ba27d8d27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42071 } }
I20260810 22:38:29.243903 8928 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9987237255af46a8af7b955ba27d8d27 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:38:29.243989 8928 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9987237255af46a8af7b955ba27d8d27 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:38:29.244117 8928 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9987237255af46a8af7b955ba27d8d27 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:38:29.245584 8928 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9987237255af46a8af7b955ba27d8d27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9987237255af46a8af7b955ba27d8d27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42071 } }
I20260810 22:38:29.246027 8928 leader_election.cc:302] T 00000000000000000000000000000000 P 9987237255af46a8af7b955ba27d8d27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9987237255af46a8af7b955ba27d8d27; no voters:
I20260810 22:38:29.246483 8928 leader_election.cc:288] T 00000000000000000000000000000000 P 9987237255af46a8af7b955ba27d8d27 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:38:29.246588 8940 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9987237255af46a8af7b955ba27d8d27 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:38:29.246862 8940 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9987237255af46a8af7b955ba27d8d27 [term 1 LEADER]: Becoming Leader. State: Replica: 9987237255af46a8af7b955ba27d8d27, State: Running, Role: LEADER
I20260810 22:38:29.247264 8940 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9987237255af46a8af7b955ba27d8d27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9987237255af46a8af7b955ba27d8d27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42071 } }
I20260810 22:38:29.247860 8932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:29.243011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.120:57913 (local address 127.0.0.1:42071)
0810 22:38:29.243143 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:29.243148 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:29.243439 (+ 291us) server_negotiation.cc:407] Connection header received
0810 22:38:29.243577 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:29.243579 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:29.243623 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:29.243710 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:29.244092 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:29.244697 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:29.245933 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:29.246194 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:29.246302 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:29.247324 (+ 1022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:29.247328 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:29.247352 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:38:29.247370 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:29.247403 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:38:29.247417 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:29.247511 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:29.247686 (+ 175us) server_negotiation.cc:299] Negotiation successful
0810 22:38:29.247724 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 22:38:29.248078 8928 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9987237255af46a8af7b955ba27d8d27 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:38:29.249751 8942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9987237255af46a8af7b955ba27d8d27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9987237255af46a8af7b955ba27d8d27. Latest consensus state: current_term: 1 leader_uuid: "9987237255af46a8af7b955ba27d8d27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9987237255af46a8af7b955ba27d8d27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42071 } } }
I20260810 22:38:29.249791 8941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9987237255af46a8af7b955ba27d8d27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9987237255af46a8af7b955ba27d8d27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9987237255af46a8af7b955ba27d8d27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42071 } } }
I20260810 22:38:29.249883 8942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9987237255af46a8af7b955ba27d8d27 [sys.catalog]: This master's current role is: LEADER
I20260810 22:38:29.249922 8941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9987237255af46a8af7b955ba27d8d27 [sys.catalog]: This master's current role is: LEADER
I20260810 22:38:29.250481 8953 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:38:29.254191 8953 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:38:29.260630 8953 catalog_manager.cc:1347] Generated new cluster ID: bc297a8e8a6a440690e631e36b9054ce
I20260810 22:38:29.260756 8953 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:38:29.283519 8953 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:38:29.285017 8953 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:38:29.300941 8953 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9987237255af46a8af7b955ba27d8d27: Generated new TSK 0
I20260810 22:38:29.302008 8953 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:38:29.316756 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 8852
W20260810 22:38:29.325430 8837 connection.cc:535] client connection to 127.0.0.1:42071 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42071): Connection reset by peer (error 104)
2026-08-10T22:38:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1570 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786401215402979-17317-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:38:29 dist-test-slave-77v9 krb5kdc[8963](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:38:29 dist-test-slave-77v9 krb5kdc[8963](info): set up 2 sockets
Aug 10 22:38:29 dist-test-slave-77v9 krb5kdc[8963](info): commencing operation
krb5kdc: starting...
W20260810 22:38:30.438082 17317 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Aug 10 22:38:30 dist-test-slave-77v9 krb5kdc[8963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:38:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:38:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786401215402979-17317-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786401215402979-17317-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 22:38:30.585125 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786401215402979-17317-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786401215402979-17317-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786401215402979-17317-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44855
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786401215402979-17317-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44855
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:38:30.704468 8977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:38:30.704830 8977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:38:30.704964 8977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:38:30.709218 8977 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:38:30.709343 8977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:38:30.709379 8977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:38:30.709412 8977 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:38:30.709445 8977 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:38:30.715095 8977 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786401215402979-17317-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786401215402979-17317-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44855
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786401215402979-17317-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786401215402979-17317-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:38:30.716370 8977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:38:30.717518 8977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:38:30.726663 8982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:30.726668 8983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:30.726724 8985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:38:30.726702 8977 server_base.cc:1034] running on GCE node
I20260810 22:38:30.727576 8977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:38:30.728790 8977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:38:30.730074 8977 hybrid_clock.cc:648] HybridClock initialized: now 1786401510730049 us; error 43 us; skew 500 ppm
Aug 10 22:38:30 dist-test-slave-77v9 krb5kdc[8963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401510, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:38:30.733119 8977 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 22:38:30.734138 8977 webserver.cc:460] Webserver started at http://127.0.0.1:34707/ using document root <none> and password file <none>
I20260810 22:38:30.734759 8977 fs_manager.cc:359] Metadata directory not provided
I20260810 22:38:30.734864 8977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:38:30.735109 8977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:38:30.737356 8977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786401215402979-17317-0/minicluster-data/master-0/data/instance:
uuid: "60ccaef2848f41f99e6cec0b6329bf18"
format_stamp: "Formatted at 2026-08-10 22:38:30 on dist-test-slave-77v9"
server_key: "7576443fe4f9300d261f36feb0385bd6"
server_key_iv: "9169915bf66e8838a2c555eb6c66acde"
server_key_version: "encryptionkey@0"
I20260810 22:38:30.737972 8977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786401215402979-17317-0/minicluster-data/master-0/wal/instance:
uuid: "60ccaef2848f41f99e6cec0b6329bf18"
format_stamp: "Formatted at 2026-08-10 22:38:30 on dist-test-slave-77v9"
server_key: "7576443fe4f9300d261f36feb0385bd6"
server_key_iv: "9169915bf66e8838a2c555eb6c66acde"
server_key_version: "encryptionkey@0"
I20260810 22:38:30.742018 8977 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 22:38:30.744728 8993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:30.746405 8977 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:38:30.746619 8977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786401215402979-17317-0/minicluster-data/master-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786401215402979-17317-0/minicluster-data/master-0/wal
uuid: "60ccaef2848f41f99e6cec0b6329bf18"
format_stamp: "Formatted at 2026-08-10 22:38:30 on dist-test-slave-77v9"
server_key: "7576443fe4f9300d261f36feb0385bd6"
server_key_iv: "9169915bf66e8838a2c555eb6c66acde"
server_key_version: "encryptionkey@0"
I20260810 22:38:30.746776 8977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786401215402979-17317-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786401215402979-17317-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786401215402979-17317-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:38:30.752339 8977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:38:30.756106 8977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:38:30.756407 8977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:38:30.764281 8977 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44855
I20260810 22:38:30.764320 9051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44855 every 8 connection(s)
I20260810 22:38:30.765412 8977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
I20260810 22:38:30.768548 9052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:38:30.773371 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 8977
I20260810 22:38:30.773497 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786401215402979-17317-0/minicluster-data/master-0/wal/instance
I20260810 22:38:30.773880 17317 external_mini_cluster.cc:1428] Setting key 5f5c6e15ced31a270c351cd49a1271fc
I20260810 22:38:30.775106 9052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60ccaef2848f41f99e6cec0b6329bf18: Bootstrap starting.
I20260810 22:38:30.777704 9052 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60ccaef2848f41f99e6cec0b6329bf18: Neither blocks nor log segments found. Creating new log.
I20260810 22:38:30.778693 9052 log.cc:824] T 00000000000000000000000000000000 P 60ccaef2848f41f99e6cec0b6329bf18: Log is configured to *not* fsync() on all Append() calls
I20260810 22:38:30.780076 9056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:30.775506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.120:36415 (local address 127.0.0.1:44855)
0810 22:38:30.775984 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:30.775999 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:30.776037 (+ 38us) server_negotiation.cc:407] Connection header received
0810 22:38:30.776774 (+ 737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:30.776800 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:30.777319 (+ 519us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:30.777711 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:30.778640 (+ 929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:30.778645 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:30.778657 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:38:30.778669 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:30.778726 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:38:30.778745 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:30.778870 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:30.778988 (+ 118us) server_negotiation.cc:299] Negotiation successful
0810 22:38:30.779163 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":138,"threads_started":1}
I20260810 22:38:30.781177 9052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60ccaef2848f41f99e6cec0b6329bf18: No bootstrap required, opened a new log
I20260810 22:38:30.783211 17317 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 22:38:30.785578 9052 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60ccaef2848f41f99e6cec0b6329bf18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ccaef2848f41f99e6cec0b6329bf18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44855 } }
I20260810 22:38:30.785931 9052 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60ccaef2848f41f99e6cec0b6329bf18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:38:30.786005 9052 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60ccaef2848f41f99e6cec0b6329bf18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ccaef2848f41f99e6cec0b6329bf18, State: Initialized, Role: FOLLOWER
I20260810 22:38:30.786517 9052 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60ccaef2848f41f99e6cec0b6329bf18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ccaef2848f41f99e6cec0b6329bf18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44855 } }
I20260810 22:38:30.786711 9052 raft_consensus.cc:388] T 00000000000000000000000000000000 P 60ccaef2848f41f99e6cec0b6329bf18 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:38:30.786800 9052 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60ccaef2848f41f99e6cec0b6329bf18 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:38:30.786967 9052 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60ccaef2848f41f99e6cec0b6329bf18 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:38:30.787465 9056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:30.785608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.120:33365 (local address 127.0.0.1:44855)
0810 22:38:30.785737 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:30.785742 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:30.785893 (+ 151us) server_negotiation.cc:407] Connection header received
0810 22:38:30.786035 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:30.786038 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:30.786086 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:30.786175 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:30.787042 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:30.787046 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:30.787049 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:38:30.787061 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:30.787093 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:38:30.787107 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:30.787190 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:30.787258 (+ 68us) server_negotiation.cc:299] Negotiation successful
0810 22:38:30.787310 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 22:38:30.788203 9052 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60ccaef2848f41f99e6cec0b6329bf18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ccaef2848f41f99e6cec0b6329bf18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44855 } }
I20260810 22:38:30.788576 9052 leader_election.cc:302] T 00000000000000000000000000000000 P 60ccaef2848f41f99e6cec0b6329bf18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60ccaef2848f41f99e6cec0b6329bf18; no voters:
I20260810 22:38:30.788929 9052 leader_election.cc:288] T 00000000000000000000000000000000 P 60ccaef2848f41f99e6cec0b6329bf18 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:38:30.789058 9064 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60ccaef2848f41f99e6cec0b6329bf18 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:38:30.789312 9064 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60ccaef2848f41f99e6cec0b6329bf18 [term 1 LEADER]: Becoming Leader. State: Replica: 60ccaef2848f41f99e6cec0b6329bf18, State: Running, Role: LEADER
I20260810 22:38:30.789638 9064 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60ccaef2848f41f99e6cec0b6329bf18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ccaef2848f41f99e6cec0b6329bf18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44855 } }
I20260810 22:38:30.790426 9052 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60ccaef2848f41f99e6cec0b6329bf18 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:38:30.791548 9066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60ccaef2848f41f99e6cec0b6329bf18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60ccaef2848f41f99e6cec0b6329bf18. Latest consensus state: current_term: 1 leader_uuid: "60ccaef2848f41f99e6cec0b6329bf18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ccaef2848f41f99e6cec0b6329bf18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44855 } } }
I20260810 22:38:30.791594 9065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60ccaef2848f41f99e6cec0b6329bf18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60ccaef2848f41f99e6cec0b6329bf18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ccaef2848f41f99e6cec0b6329bf18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44855 } } }
I20260810 22:38:30.791680 9066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60ccaef2848f41f99e6cec0b6329bf18 [sys.catalog]: This master's current role is: LEADER
I20260810 22:38:30.791693 9065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60ccaef2848f41f99e6cec0b6329bf18 [sys.catalog]: This master's current role is: LEADER
I20260810 22:38:30.792062 9073 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:38:30.794906 9073 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:38:30.800932 9073 catalog_manager.cc:1347] Generated new cluster ID: 65fc39ae67ee401088b5d834d53b8c5b
I20260810 22:38:30.801044 9073 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:38:30.814993 9073 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:38:30.815361 9073 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:38:30.823657 9073 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 60ccaef2848f41f99e6cec0b6329bf18: Generated new TSK 0
I20260810 22:38:30.824455 9073 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:38:30.830516 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 8977
2026-08-10T22:38:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1512 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786401215402979-17317-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:38:30 dist-test-slave-77v9 krb5kdc[9087](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:38:30 dist-test-slave-77v9 krb5kdc[9087](info): set up 2 sockets
Aug 10 22:38:30 dist-test-slave-77v9 krb5kdc[9087](info): commencing operation
krb5kdc: starting...
W20260810 22:38:32.000139 17317 mini_kdc.cc:121] Time spent starting KDC: real 1.147s user 0.005s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Aug 10 22:38:32 dist-test-slave-77v9 krb5kdc[9087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:38:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:38:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.233.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab.
Entry for principal kudu/127.16.233.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab.
I20260810 22:38:32.149017 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786401215402979-17317-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786401215402979-17317-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786401215402979-17317-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:36703
--webserver_interface=127.16.233.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab
--principal=kudu/127.16.233.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:44917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.126:36703
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:38:32.265578 9102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:38:32.265877 9102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:38:32.265924 9102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:38:32.270117 9102 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:38:32.270193 9102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:38:32.270213 9102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:38:32.270231 9102 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:38:32.270246 9102 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:38:32.275765 9102 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:44917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786401215402979-17317-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786401215402979-17317-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.126:36703
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab
--principal=kudu/127.16.233.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:36703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786401215402979-17317-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:38:32.277007 9102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:38:32.278070 9102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:38:32.287092 9111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:32.287060 9109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:32.287139 9107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:38:32.287386 9102 server_base.cc:1034] running on GCE node
I20260810 22:38:32.288086 9102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:38:32.289379 9102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:38:32.290547 9102 hybrid_clock.cc:648] HybridClock initialized: now 1786401512290493 us; error 73 us; skew 500 ppm
Aug 10 22:38:32 dist-test-slave-77v9 krb5kdc[9087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401512, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:38:32.293466 9102 init.cc:377] Logged in from keytab as kudu/127.16.233.126@KRBTEST.COM (short username kudu)
I20260810 22:38:32.294643 9102 webserver.cc:460] Webserver started at http://127.16.233.126:43769/ using document root <none> and password file <none>
I20260810 22:38:32.295313 9102 fs_manager.cc:359] Metadata directory not provided
I20260810 22:38:32.295423 9102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:38:32.295696 9102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:38:32.298120 9102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786401215402979-17317-0/minicluster-data/master-0/data/instance:
uuid: "e9cfe5bb7b344fd5a55c4cb823eda004"
format_stamp: "Formatted at 2026-08-10 22:38:32 on dist-test-slave-77v9"
server_key: "6a866118c18d057e1cc8b20f50d601ff"
server_key_iv: "3607fb3b32bd24ddff51c28eb4ad80c7"
server_key_version: "encryptionkey@0"
I20260810 22:38:32.298797 9102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786401215402979-17317-0/minicluster-data/master-0/wal/instance:
uuid: "e9cfe5bb7b344fd5a55c4cb823eda004"
format_stamp: "Formatted at 2026-08-10 22:38:32 on dist-test-slave-77v9"
server_key: "6a866118c18d057e1cc8b20f50d601ff"
server_key_iv: "3607fb3b32bd24ddff51c28eb4ad80c7"
server_key_version: "encryptionkey@0"
I20260810 22:38:32.303261 9102 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 22:38:32.306173 9120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:32.307672 9102 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 22:38:32.307801 9102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786401215402979-17317-0/minicluster-data/master-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786401215402979-17317-0/minicluster-data/master-0/wal
uuid: "e9cfe5bb7b344fd5a55c4cb823eda004"
format_stamp: "Formatted at 2026-08-10 22:38:32 on dist-test-slave-77v9"
server_key: "6a866118c18d057e1cc8b20f50d601ff"
server_key_iv: "3607fb3b32bd24ddff51c28eb4ad80c7"
server_key_version: "encryptionkey@0"
I20260810 22:38:32.307900 9102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786401215402979-17317-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786401215402979-17317-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786401215402979-17317-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:38:32.328522 9102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:38:32.332350 9102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:38:32.332553 9102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:38:32.340145 9102 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.126:36703
I20260810 22:38:32.340230 9175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.126:36703 every 8 connection(s)
I20260810 22:38:32.341288 9102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
I20260810 22:38:32.344194 9176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:38:32.347654 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 9102
I20260810 22:38:32.347869 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786401215402979-17317-0/minicluster-data/master-0/wal/instance
I20260810 22:38:32.348191 17317 external_mini_cluster.cc:1428] Setting key 40ac4b32eba72f5436e298257afc2bd5
I20260810 22:38:32.350778 9176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9cfe5bb7b344fd5a55c4cb823eda004: Bootstrap starting.
I20260810 22:38:32.353534 9176 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9cfe5bb7b344fd5a55c4cb823eda004: Neither blocks nor log segments found. Creating new log.
I20260810 22:38:32.354480 9176 log.cc:824] T 00000000000000000000000000000000 P e9cfe5bb7b344fd5a55c4cb823eda004: Log is configured to *not* fsync() on all Append() calls
I20260810 22:38:32.356979 9176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9cfe5bb7b344fd5a55c4cb823eda004: No bootstrap required, opened a new log
Aug 10 22:38:32 dist-test-slave-77v9 krb5kdc[9087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:38:32.360905 9176 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9cfe5bb7b344fd5a55c4cb823eda004 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9cfe5bb7b344fd5a55c4cb823eda004" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 36703 } }
I20260810 22:38:32.361181 9176 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9cfe5bb7b344fd5a55c4cb823eda004 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:38:32.361240 9176 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9cfe5bb7b344fd5a55c4cb823eda004 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9cfe5bb7b344fd5a55c4cb823eda004, State: Initialized, Role: FOLLOWER
I20260810 22:38:32.361773 9176 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9cfe5bb7b344fd5a55c4cb823eda004 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9cfe5bb7b344fd5a55c4cb823eda004" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 36703 } }
I20260810 22:38:32.361934 9176 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9cfe5bb7b344fd5a55c4cb823eda004 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:38:32.362025 9176 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9cfe5bb7b344fd5a55c4cb823eda004 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:38:32.362124 9176 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9cfe5bb7b344fd5a55c4cb823eda004 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:38:32.363197 9176 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9cfe5bb7b344fd5a55c4cb823eda004 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9cfe5bb7b344fd5a55c4cb823eda004" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 36703 } }
I20260810 22:38:32.363562 9176 leader_election.cc:302] T 00000000000000000000000000000000 P e9cfe5bb7b344fd5a55c4cb823eda004 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9cfe5bb7b344fd5a55c4cb823eda004; no voters:
I20260810 22:38:32.363924 9176 leader_election.cc:288] T 00000000000000000000000000000000 P e9cfe5bb7b344fd5a55c4cb823eda004 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:38:32.364012 9181 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9cfe5bb7b344fd5a55c4cb823eda004 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:38:32.364179 9181 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9cfe5bb7b344fd5a55c4cb823eda004 [term 1 LEADER]: Becoming Leader. State: Replica: e9cfe5bb7b344fd5a55c4cb823eda004, State: Running, Role: LEADER
I20260810 22:38:32.364421 9179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:32.349788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58505 (local address 127.16.233.126:36703)
0810 22:38:32.350266 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:32.350284 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:32.350324 (+ 40us) server_negotiation.cc:407] Connection header received
0810 22:38:32.351121 (+ 797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:32.351140 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:32.351506 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:32.351809 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:32.352870 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:32.354053 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:32.355236 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:32.355785 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:32.355917 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:32.359025 (+ 3108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:32.359031 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:32.359045 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:38:32.359075 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:32.361030 (+ 1955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:32.361631 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:32.361636 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:32.361641 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:32.361754 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:32.362109 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:32.362113 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:32.362114 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:32.362572 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:32.362821 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:32.363133 (+ 312us) server_negotiation.cc:299] Negotiation successful
0810 22:38:32.363417 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":155,"threads_started":1}
I20260810 22:38:32.364529 9181 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9cfe5bb7b344fd5a55c4cb823eda004 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9cfe5bb7b344fd5a55c4cb823eda004" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 36703 } }
I20260810 22:38:32.365262 9176 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9cfe5bb7b344fd5a55c4cb823eda004 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:38:32.365728 9181 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9cfe5bb7b344fd5a55c4cb823eda004 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9cfe5bb7b344fd5a55c4cb823eda004. Latest consensus state: current_term: 1 leader_uuid: "e9cfe5bb7b344fd5a55c4cb823eda004" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9cfe5bb7b344fd5a55c4cb823eda004" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 36703 } } }
I20260810 22:38:32.365916 9181 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9cfe5bb7b344fd5a55c4cb823eda004 [sys.catalog]: This master's current role is: LEADER
I20260810 22:38:32.366147 9182 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9cfe5bb7b344fd5a55c4cb823eda004 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9cfe5bb7b344fd5a55c4cb823eda004" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9cfe5bb7b344fd5a55c4cb823eda004" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 36703 } } }
I20260810 22:38:32.366292 9182 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9cfe5bb7b344fd5a55c4cb823eda004 [sys.catalog]: This master's current role is: LEADER
I20260810 22:38:32.366418 9186 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:38:32.367746 17317 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 22:38:32.372339 9186 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:38:32.378741 9186 catalog_manager.cc:1347] Generated new cluster ID: 66c8b25c967f4a06875877dc18ebb199
I20260810 22:38:32.378834 9186 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:38:32.379933 9179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:32.370190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40299 (local address 127.16.233.126:36703)
0810 22:38:32.370318 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:32.370368 (+ 50us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:32.370503 (+ 135us) server_negotiation.cc:407] Connection header received
0810 22:38:32.370652 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:32.370654 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:32.370708 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:32.370791 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:32.371615 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:32.372402 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:32.373556 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:32.373803 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:32.373919 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:32.375656 (+ 1737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:32.375660 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:32.375663 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:38:32.375689 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:32.377713 (+ 2024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:32.378311 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:32.378315 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:32.378317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:32.378433 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:32.378884 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:32.378887 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:32.378889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:32.379107 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:32.379265 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:32.379681 (+ 416us) server_negotiation.cc:299] Negotiation successful
0810 22:38:32.379783 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 22:38:32.392431 9186 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:38:32.393761 9186 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:38:32.402171 9186 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9cfe5bb7b344fd5a55c4cb823eda004: Generated new TSK 0
I20260810 22:38:32.402901 9186 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:38:32.412516 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 9102
W20260810 22:38:32.420190 9084 connection.cc:535] client connection to 127.16.233.126:36703 recv error: Network error: failed to read from TLS socket (remote: 127.16.233.126:36703): Connection reset by peer (error 104)
2026-08-10T22:38:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1581 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786401215402979-17317-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:38:32 dist-test-slave-77v9 krb5kdc[9209](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:38:32 dist-test-slave-77v9 krb5kdc[9209](info): set up 2 sockets
Aug 10 22:38:32 dist-test-slave-77v9 krb5kdc[9209](info): commencing operation
krb5kdc: starting...
W20260810 22:38:33.581329 17317 mini_kdc.cc:121] Time spent starting KDC: real 1.146s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Aug 10 22:38:33 dist-test-slave-77v9 krb5kdc[9209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:38:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:38:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.233.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab.
Entry for principal kudu/127.16.233.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab.
I20260810 22:38:33.733220 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786401215402979-17317-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786401215402979-17317-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786401215402979-17317-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:39585
--webserver_interface=127.16.233.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab
--principal=kudu/127.16.233.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:39693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.126:39585
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:38:33.858753 9224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:38:33.859098 9224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:38:33.859153 9224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:38:33.863512 9224 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:38:33.863615 9224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:38:33.863638 9224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:38:33.863658 9224 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:38:33.863678 9224 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:38:33.869865 9224 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:39693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786401215402979-17317-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786401215402979-17317-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.126:39585
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab
--principal=kudu/127.16.233.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:39585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9224
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786401215402979-17317-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:38:33.871167 9224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:38:33.872308 9224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:38:33.881840 9232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:33.881881 9229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:33.881881 9230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:38:33.882290 9224 server_base.cc:1034] running on GCE node
I20260810 22:38:33.882925 9224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:38:33.884301 9224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:38:33.885524 9224 hybrid_clock.cc:648] HybridClock initialized: now 1786401513885499 us; error 40 us; skew 500 ppm
Aug 10 22:38:33 dist-test-slave-77v9 krb5kdc[9209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401513, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:38:33.893457 9224 init.cc:377] Logged in from keytab as kudu/127.16.233.126@KRBTEST.COM (short username kudu)
I20260810 22:38:33.894675 9224 webserver.cc:460] Webserver started at http://127.16.233.126:43051/ using document root <none> and password file <none>
I20260810 22:38:33.895305 9224 fs_manager.cc:359] Metadata directory not provided
I20260810 22:38:33.895378 9224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:38:33.895632 9224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:38:33.898067 9224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786401215402979-17317-0/minicluster-data/master-0/data/instance:
uuid: "348adc77ab93438d961ab05e119cb8e2"
format_stamp: "Formatted at 2026-08-10 22:38:33 on dist-test-slave-77v9"
server_key: "e4379d091a0f1f2ddb376cedbb30a2d2"
server_key_iv: "1d03b26099acea90f18750194789e6d5"
server_key_version: "encryptionkey@0"
I20260810 22:38:33.898725 9224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786401215402979-17317-0/minicluster-data/master-0/wal/instance:
uuid: "348adc77ab93438d961ab05e119cb8e2"
format_stamp: "Formatted at 2026-08-10 22:38:33 on dist-test-slave-77v9"
server_key: "e4379d091a0f1f2ddb376cedbb30a2d2"
server_key_iv: "1d03b26099acea90f18750194789e6d5"
server_key_version: "encryptionkey@0"
I20260810 22:38:33.903110 9224 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 22:38:33.905907 9240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:33.907418 9224 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:38:33.907549 9224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786401215402979-17317-0/minicluster-data/master-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786401215402979-17317-0/minicluster-data/master-0/wal
uuid: "348adc77ab93438d961ab05e119cb8e2"
format_stamp: "Formatted at 2026-08-10 22:38:33 on dist-test-slave-77v9"
server_key: "e4379d091a0f1f2ddb376cedbb30a2d2"
server_key_iv: "1d03b26099acea90f18750194789e6d5"
server_key_version: "encryptionkey@0"
I20260810 22:38:33.907724 9224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786401215402979-17317-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786401215402979-17317-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786401215402979-17317-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:38:33.922883 9224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:38:33.926659 9224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:38:33.926900 9224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:38:33.935155 9224 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.126:39585
I20260810 22:38:33.935182 9294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.126:39585 every 8 connection(s)
I20260810 22:38:33.936255 9224 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
I20260810 22:38:33.939273 9295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:38:33.942461 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 9224
I20260810 22:38:33.942646 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786401215402979-17317-0/minicluster-data/master-0/wal/instance
I20260810 22:38:33.942975 17317 external_mini_cluster.cc:1428] Setting key ce1db72330253507f11d46c7911a88f8
I20260810 22:38:33.945932 9295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 348adc77ab93438d961ab05e119cb8e2: Bootstrap starting.
I20260810 22:38:33.948292 9295 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 348adc77ab93438d961ab05e119cb8e2: Neither blocks nor log segments found. Creating new log.
I20260810 22:38:33.949155 9295 log.cc:824] T 00000000000000000000000000000000 P 348adc77ab93438d961ab05e119cb8e2: Log is configured to *not* fsync() on all Append() calls
I20260810 22:38:33.951659 9295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 348adc77ab93438d961ab05e119cb8e2: No bootstrap required, opened a new log
I20260810 22:38:33.952950 9299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:33.944525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33971 (local address 127.16.233.126:39585)
0810 22:38:33.945074 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:33.945095 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:33.945134 (+ 39us) server_negotiation.cc:407] Connection header received
0810 22:38:33.945786 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:33.945803 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:33.946221 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:33.946642 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:33.947075 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:33.948088 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:33.949159 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:33.949824 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:33.950004 (+ 180us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:33.951053 (+ 1049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:33.951058 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:33.951072 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:38:33.951491 (+ 419us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:33.951550 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:38:33.951572 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:33.951678 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:33.951903 (+ 225us) server_negotiation.cc:299] Negotiation successful
0810 22:38:33.952078 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":211,"threads_started":1}
I20260810 22:38:33.956223 17317 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 22:38:33.956142 9295 raft_consensus.cc:348] T 00000000000000000000000000000000 P 348adc77ab93438d961ab05e119cb8e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "348adc77ab93438d961ab05e119cb8e2" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39585 } }
I20260810 22:38:33.956446 9295 raft_consensus.cc:374] T 00000000000000000000000000000000 P 348adc77ab93438d961ab05e119cb8e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:38:33.956525 9295 raft_consensus.cc:729] T 00000000000000000000000000000000 P 348adc77ab93438d961ab05e119cb8e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 348adc77ab93438d961ab05e119cb8e2, State: Initialized, Role: FOLLOWER
I20260810 22:38:33.957003 9295 consensus_queue.cc:260] T 00000000000000000000000000000000 P 348adc77ab93438d961ab05e119cb8e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "348adc77ab93438d961ab05e119cb8e2" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39585 } }
I20260810 22:38:33.957163 9295 raft_consensus.cc:388] T 00000000000000000000000000000000 P 348adc77ab93438d961ab05e119cb8e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:38:33.957252 9295 raft_consensus.cc:482] T 00000000000000000000000000000000 P 348adc77ab93438d961ab05e119cb8e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:38:33.957372 9295 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 348adc77ab93438d961ab05e119cb8e2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:38:33.958530 9295 raft_consensus.cc:504] T 00000000000000000000000000000000 P 348adc77ab93438d961ab05e119cb8e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "348adc77ab93438d961ab05e119cb8e2" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39585 } }
I20260810 22:38:33.958935 9295 leader_election.cc:302] T 00000000000000000000000000000000 P 348adc77ab93438d961ab05e119cb8e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 348adc77ab93438d961ab05e119cb8e2; no voters:
I20260810 22:38:33.959291 9295 leader_election.cc:288] T 00000000000000000000000000000000 P 348adc77ab93438d961ab05e119cb8e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:38:33.959401 9306 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 348adc77ab93438d961ab05e119cb8e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:38:33.959592 9306 raft_consensus.cc:686] T 00000000000000000000000000000000 P 348adc77ab93438d961ab05e119cb8e2 [term 1 LEADER]: Becoming Leader. State: Replica: 348adc77ab93438d961ab05e119cb8e2, State: Running, Role: LEADER
I20260810 22:38:33.959965 9306 consensus_queue.cc:237] T 00000000000000000000000000000000 P 348adc77ab93438d961ab05e119cb8e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "348adc77ab93438d961ab05e119cb8e2" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39585 } }
I20260810 22:38:33.960688 9295 sys_catalog.cc:564] T 00000000000000000000000000000000 P 348adc77ab93438d961ab05e119cb8e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:38:33.961859 9308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 348adc77ab93438d961ab05e119cb8e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 348adc77ab93438d961ab05e119cb8e2. Latest consensus state: current_term: 1 leader_uuid: "348adc77ab93438d961ab05e119cb8e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "348adc77ab93438d961ab05e119cb8e2" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39585 } } }
I20260810 22:38:33.962023 9308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 348adc77ab93438d961ab05e119cb8e2 [sys.catalog]: This master's current role is: LEADER
I20260810 22:38:33.962396 9307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 348adc77ab93438d961ab05e119cb8e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "348adc77ab93438d961ab05e119cb8e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "348adc77ab93438d961ab05e119cb8e2" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39585 } } }
I20260810 22:38:33.962473 9315 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:38:33.962492 9307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 348adc77ab93438d961ab05e119cb8e2 [sys.catalog]: This master's current role is: LEADER
I20260810 22:38:33.963738 9299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:33.958748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56571 (local address 127.16.233.126:39585)
0810 22:38:33.958884 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:33.958888 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:33.959098 (+ 210us) server_negotiation.cc:407] Connection header received
0810 22:38:33.959276 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:33.959279 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:33.959332 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:33.959425 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:33.959884 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:33.960480 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:33.961574 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:33.961941 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:33.962058 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:33.963221 (+ 1163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:33.963226 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:33.963228 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:38:33.963242 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:33.963274 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:38:33.963288 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:33.963383 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:33.963571 (+ 188us) server_negotiation.cc:299] Negotiation successful
0810 22:38:33.963609 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 22:38:33.966068 9315 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:38:33.971966 9315 catalog_manager.cc:1347] Generated new cluster ID: 7089d1b6978f423888a2b3a331bf7a1e
I20260810 22:38:33.972054 9315 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:38:33.980036 9315 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:38:33.981467 9315 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:38:33.988726 9315 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 348adc77ab93438d961ab05e119cb8e2: Generated new TSK 0
I20260810 22:38:33.989629 9315 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:38:33.999150 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 9224
W20260810 22:38:34.007979 9207 connection.cc:535] client connection to 127.16.233.126:39585 recv error: Network error: failed to read from TLS socket (remote: 127.16.233.126:39585): Connection reset by peer (error 104)
2026-08-10T22:38:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1587 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786401215402979-17317-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:38:34 dist-test-slave-77v9 krb5kdc[9327](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:38:34 dist-test-slave-77v9 krb5kdc[9327](info): set up 2 sockets
Aug 10 22:38:34 dist-test-slave-77v9 krb5kdc[9327](info): commencing operation
krb5kdc: starting...
W20260810 22:38:35.151922 17317 mini_kdc.cc:121] Time spent starting KDC: real 1.129s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Aug 10 22:38:35 dist-test-slave-77v9 krb5kdc[9327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:38:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:38:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.233.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab.
Entry for principal kudu/127.16.233.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab.
I20260810 22:38:35.300992 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786401215402979-17317-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786401215402979-17317-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786401215402979-17317-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:39709
--webserver_interface=127.16.233.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab
--principal=kudu/127.16.233.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:41885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.126:39709
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:38:35.426787 9342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:38:35.427261 9342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:38:35.427358 9342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:38:35.435566 9342 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:38:35.435703 9342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:38:35.435753 9342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:38:35.435796 9342 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:38:35.435827 9342 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:38:35.442672 9342 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:41885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786401215402979-17317-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786401215402979-17317-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.126:39709
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab
--principal=kudu/127.16.233.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:39709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786401215402979-17317-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:38:35.443974 9342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:38:35.445343 9342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:38:35.455015 9350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:35.455034 9349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:35.455037 9352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:38:35.455188 9342 server_base.cc:1034] running on GCE node
I20260810 22:38:35.456123 9342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:38:35.457433 9342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:38:35.458635 9342 hybrid_clock.cc:648] HybridClock initialized: now 1786401515458620 us; error 41 us; skew 500 ppm
Aug 10 22:38:35 dist-test-slave-77v9 krb5kdc[9327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401515, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:38:35.462267 9342 init.cc:377] Logged in from keytab as kudu/127.16.233.126@KRBTEST.COM (short username kudu)
I20260810 22:38:35.463351 9342 webserver.cc:460] Webserver started at http://127.16.233.126:36275/ using document root <none> and password file <none>
I20260810 22:38:35.463982 9342 fs_manager.cc:359] Metadata directory not provided
I20260810 22:38:35.464087 9342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:38:35.464337 9342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:38:35.466756 9342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786401215402979-17317-0/minicluster-data/master-0/data/instance:
uuid: "a60c4ae6672e414a8c056f5c14a1f371"
format_stamp: "Formatted at 2026-08-10 22:38:35 on dist-test-slave-77v9"
server_key: "ffc59ca4689a6ae0067cf62f283eca21"
server_key_iv: "940e3f6b20026967f7d16b42d16d757b"
server_key_version: "encryptionkey@0"
I20260810 22:38:35.467422 9342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786401215402979-17317-0/minicluster-data/master-0/wal/instance:
uuid: "a60c4ae6672e414a8c056f5c14a1f371"
format_stamp: "Formatted at 2026-08-10 22:38:35 on dist-test-slave-77v9"
server_key: "ffc59ca4689a6ae0067cf62f283eca21"
server_key_iv: "940e3f6b20026967f7d16b42d16d757b"
server_key_version: "encryptionkey@0"
I20260810 22:38:35.471581 9342 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260810 22:38:35.474278 9359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:35.475706 9342 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 22:38:35.475894 9342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786401215402979-17317-0/minicluster-data/master-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786401215402979-17317-0/minicluster-data/master-0/wal
uuid: "a60c4ae6672e414a8c056f5c14a1f371"
format_stamp: "Formatted at 2026-08-10 22:38:35 on dist-test-slave-77v9"
server_key: "ffc59ca4689a6ae0067cf62f283eca21"
server_key_iv: "940e3f6b20026967f7d16b42d16d757b"
server_key_version: "encryptionkey@0"
I20260810 22:38:35.476027 9342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786401215402979-17317-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786401215402979-17317-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786401215402979-17317-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:38:35.481027 9342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:38:35.484654 9342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:38:35.484941 9342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:38:35.492758 9342 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.126:39709
I20260810 22:38:35.492941 9417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.126:39709 every 8 connection(s)
I20260810 22:38:35.493891 9342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
I20260810 22:38:35.497185 9418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:38:35.499745 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 9342
I20260810 22:38:35.499912 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786401215402979-17317-0/minicluster-data/master-0/wal/instance
I20260810 22:38:35.500258 17317 external_mini_cluster.cc:1428] Setting key d5efb68e42b040ca2c56dc050214e00b
I20260810 22:38:35.504485 9418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a60c4ae6672e414a8c056f5c14a1f371: Bootstrap starting.
I20260810 22:38:35.506338 9421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:35.501947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53747 (local address 127.16.233.126:39709)
0810 22:38:35.502471 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:35.502487 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:35.502524 (+ 37us) server_negotiation.cc:407] Connection header received
0810 22:38:35.503484 (+ 960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:35.503505 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:35.503817 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:35.504180 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:35.505124 (+ 944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:35.505130 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:35.505141 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:38:35.505152 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:35.505206 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:38:35.505225 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:35.505386 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:35.505458 (+ 72us) server_negotiation.cc:299] Negotiation successful
0810 22:38:35.505612 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":172,"threads_started":1}
I20260810 22:38:35.507302 9418 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a60c4ae6672e414a8c056f5c14a1f371: Neither blocks nor log segments found. Creating new log.
I20260810 22:38:35.508189 9418 log.cc:824] T 00000000000000000000000000000000 P a60c4ae6672e414a8c056f5c14a1f371: Log is configured to *not* fsync() on all Append() calls
I20260810 22:38:35.509500 17317 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 22:38:35.510644 9418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a60c4ae6672e414a8c056f5c14a1f371: No bootstrap required, opened a new log
I20260810 22:38:35.513803 9421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:35.511732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45519 (local address 127.16.233.126:39709)
0810 22:38:35.511894 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:35.511901 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:35.512124 (+ 223us) server_negotiation.cc:407] Connection header received
0810 22:38:35.512320 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:35.512324 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:35.512366 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:35.512463 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:35.513308 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:35.513311 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:35.513314 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:38:35.513327 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:35.513360 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:38:35.513378 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:35.513482 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:35.513610 (+ 128us) server_negotiation.cc:299] Negotiation successful
0810 22:38:35.513656 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 22:38:35.515554 9418 raft_consensus.cc:348] T 00000000000000000000000000000000 P a60c4ae6672e414a8c056f5c14a1f371 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a60c4ae6672e414a8c056f5c14a1f371" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39709 } }
I20260810 22:38:35.515834 9418 raft_consensus.cc:374] T 00000000000000000000000000000000 P a60c4ae6672e414a8c056f5c14a1f371 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:38:35.515916 9418 raft_consensus.cc:729] T 00000000000000000000000000000000 P a60c4ae6672e414a8c056f5c14a1f371 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a60c4ae6672e414a8c056f5c14a1f371, State: Initialized, Role: FOLLOWER
I20260810 22:38:35.516399 9418 consensus_queue.cc:260] T 00000000000000000000000000000000 P a60c4ae6672e414a8c056f5c14a1f371 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a60c4ae6672e414a8c056f5c14a1f371" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39709 } }
I20260810 22:38:35.516561 9418 raft_consensus.cc:388] T 00000000000000000000000000000000 P a60c4ae6672e414a8c056f5c14a1f371 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:38:35.516659 9418 raft_consensus.cc:482] T 00000000000000000000000000000000 P a60c4ae6672e414a8c056f5c14a1f371 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:38:35.516765 9418 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a60c4ae6672e414a8c056f5c14a1f371 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:38:35.517855 9418 raft_consensus.cc:504] T 00000000000000000000000000000000 P a60c4ae6672e414a8c056f5c14a1f371 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a60c4ae6672e414a8c056f5c14a1f371" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39709 } }
I20260810 22:38:35.518262 9418 leader_election.cc:302] T 00000000000000000000000000000000 P a60c4ae6672e414a8c056f5c14a1f371 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a60c4ae6672e414a8c056f5c14a1f371; no voters:
I20260810 22:38:35.518611 9418 leader_election.cc:288] T 00000000000000000000000000000000 P a60c4ae6672e414a8c056f5c14a1f371 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:38:35.518870 9428 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a60c4ae6672e414a8c056f5c14a1f371 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:38:35.519150 9428 raft_consensus.cc:686] T 00000000000000000000000000000000 P a60c4ae6672e414a8c056f5c14a1f371 [term 1 LEADER]: Becoming Leader. State: Replica: a60c4ae6672e414a8c056f5c14a1f371, State: Running, Role: LEADER
I20260810 22:38:35.519533 9428 consensus_queue.cc:237] T 00000000000000000000000000000000 P a60c4ae6672e414a8c056f5c14a1f371 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a60c4ae6672e414a8c056f5c14a1f371" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39709 } }
I20260810 22:38:35.520097 9418 sys_catalog.cc:564] T 00000000000000000000000000000000 P a60c4ae6672e414a8c056f5c14a1f371 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:38:35.521557 9429 sys_catalog.cc:455] T 00000000000000000000000000000000 P a60c4ae6672e414a8c056f5c14a1f371 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a60c4ae6672e414a8c056f5c14a1f371" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a60c4ae6672e414a8c056f5c14a1f371" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39709 } } }
I20260810 22:38:35.521622 9430 sys_catalog.cc:455] T 00000000000000000000000000000000 P a60c4ae6672e414a8c056f5c14a1f371 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a60c4ae6672e414a8c056f5c14a1f371. Latest consensus state: current_term: 1 leader_uuid: "a60c4ae6672e414a8c056f5c14a1f371" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a60c4ae6672e414a8c056f5c14a1f371" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 39709 } } }
I20260810 22:38:35.521762 9430 sys_catalog.cc:458] T 00000000000000000000000000000000 P a60c4ae6672e414a8c056f5c14a1f371 [sys.catalog]: This master's current role is: LEADER
I20260810 22:38:35.521699 9429 sys_catalog.cc:458] T 00000000000000000000000000000000 P a60c4ae6672e414a8c056f5c14a1f371 [sys.catalog]: This master's current role is: LEADER
I20260810 22:38:35.522207 9439 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:38:35.525324 9439 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:38:35.530972 9439 catalog_manager.cc:1347] Generated new cluster ID: 7d9aa4a2be9446a1a5514d5266bfab0a
I20260810 22:38:35.531076 9439 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:38:35.548159 9439 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:38:35.548619 9439 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:38:35.554929 9439 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a60c4ae6672e414a8c056f5c14a1f371: Generated new TSK 0
I20260810 22:38:35.555733 9439 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:38:35.562633 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 9342
2026-08-10T22:38:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1563 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786401215402979-17317-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:38:35 dist-test-slave-77v9 krb5kdc[9451](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:38:35 dist-test-slave-77v9 krb5kdc[9451](info): set up 2 sockets
Aug 10 22:38:35 dist-test-slave-77v9 krb5kdc[9451](info): commencing operation
krb5kdc: starting...
W20260810 22:38:36.677058 17317 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.005s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Aug 10 22:38:36 dist-test-slave-77v9 krb5kdc[9451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:38:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:38:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.233.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab.
Entry for principal kudu/127.16.233.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab.
I20260810 22:38:36.825320 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786401215402979-17317-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786401215402979-17317-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786401215402979-17317-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:41843
--webserver_interface=127.16.233.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab
--principal=kudu/127.16.233.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:38287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.126:41843
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:38:36.950663 9464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:38:36.951064 9464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:38:36.951164 9464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:38:36.955677 9464 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:38:36.955790 9464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:38:36.955832 9464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:38:36.955891 9464 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:38:36.955931 9464 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:38:36.961778 9464 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:38287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786401215402979-17317-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786401215402979-17317-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.126:41843
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab
--principal=kudu/127.16.233.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:41843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786401215402979-17317-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:38:36.963150 9464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:38:36.964332 9464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:38:36.973917 9470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:36.973918 9469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:36.973908 9472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:38:36.974148 9464 server_base.cc:1034] running on GCE node
I20260810 22:38:36.974953 9464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:38:36.976276 9464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:38:36.977512 9464 hybrid_clock.cc:648] HybridClock initialized: now 1786401516977480 us; error 46 us; skew 500 ppm
Aug 10 22:38:36 dist-test-slave-77v9 krb5kdc[9451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401516, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:38:36.980567 9464 init.cc:377] Logged in from keytab as kudu/127.16.233.126@KRBTEST.COM (short username kudu)
I20260810 22:38:36.981643 9464 webserver.cc:460] Webserver started at http://127.16.233.126:43653/ using document root <none> and password file <none>
I20260810 22:38:36.982203 9464 fs_manager.cc:359] Metadata directory not provided
I20260810 22:38:36.982265 9464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:38:36.982475 9464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:38:36.984689 9464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786401215402979-17317-0/minicluster-data/master-0/data/instance:
uuid: "fbd4515f6a7d4285ae67b0841e4def9f"
format_stamp: "Formatted at 2026-08-10 22:38:36 on dist-test-slave-77v9"
server_key: "2764a3864b76d070405d8c0fc9912af5"
server_key_iv: "741b1d35b6b45494c97e0ac9f6527657"
server_key_version: "encryptionkey@0"
I20260810 22:38:36.985375 9464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786401215402979-17317-0/minicluster-data/master-0/wal/instance:
uuid: "fbd4515f6a7d4285ae67b0841e4def9f"
format_stamp: "Formatted at 2026-08-10 22:38:36 on dist-test-slave-77v9"
server_key: "2764a3864b76d070405d8c0fc9912af5"
server_key_iv: "741b1d35b6b45494c97e0ac9f6527657"
server_key_version: "encryptionkey@0"
I20260810 22:38:36.989576 9464 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 22:38:36.992276 9479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:36.993743 9464 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:38:36.993865 9464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786401215402979-17317-0/minicluster-data/master-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786401215402979-17317-0/minicluster-data/master-0/wal
uuid: "fbd4515f6a7d4285ae67b0841e4def9f"
format_stamp: "Formatted at 2026-08-10 22:38:36 on dist-test-slave-77v9"
server_key: "2764a3864b76d070405d8c0fc9912af5"
server_key_iv: "741b1d35b6b45494c97e0ac9f6527657"
server_key_version: "encryptionkey@0"
I20260810 22:38:36.993963 9464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786401215402979-17317-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786401215402979-17317-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786401215402979-17317-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:38:37.006486 9464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:38:37.009941 9464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:38:37.010120 9464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:38:37.018057 9464 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.126:41843
I20260810 22:38:37.018081 9537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.126:41843 every 8 connection(s)
I20260810 22:38:37.019102 9464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
I20260810 22:38:37.022313 9539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:38:37.024180 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 9464
I20260810 22:38:37.024328 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786401215402979-17317-0/minicluster-data/master-0/wal/instance
I20260810 22:38:37.024668 17317 external_mini_cluster.cc:1428] Setting key 0d4e89ac615cfa5a6a77a625e3bb00df
I20260810 22:38:37.030450 9539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbd4515f6a7d4285ae67b0841e4def9f: Bootstrap starting.
I20260810 22:38:37.033843 9539 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fbd4515f6a7d4285ae67b0841e4def9f: Neither blocks nor log segments found. Creating new log.
I20260810 22:38:37.034897 9539 log.cc:824] T 00000000000000000000000000000000 P fbd4515f6a7d4285ae67b0841e4def9f: Log is configured to *not* fsync() on all Append() calls
Aug 10 22:38:37 dist-test-slave-77v9 krb5kdc[9451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:38:37.037814 9539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbd4515f6a7d4285ae67b0841e4def9f: No bootstrap required, opened a new log
I20260810 22:38:37.043576 9539 raft_consensus.cc:348] T 00000000000000000000000000000000 P fbd4515f6a7d4285ae67b0841e4def9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbd4515f6a7d4285ae67b0841e4def9f" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 41843 } }
I20260810 22:38:37.043900 9542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:37.026411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42383 (local address 127.16.233.126:41843)
0810 22:38:37.026965 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:37.026990 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:37.027047 (+ 57us) server_negotiation.cc:407] Connection header received
0810 22:38:37.028071 (+ 1024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:37.028102 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:37.028634 (+ 532us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:37.029131 (+ 497us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:37.030007 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:37.031674 (+ 1667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:37.032795 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:37.033629 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:37.033853 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:37.037335 (+ 3482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:37.037343 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:37.037362 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:38:37.037404 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:37.039779 (+ 2375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:37.040289 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:37.040298 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:37.040308 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:37.040492 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:37.041047 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:37.041053 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:37.041055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:37.041692 (+ 637us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:37.041918 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:37.042237 (+ 319us) server_negotiation.cc:299] Negotiation successful
0810 22:38:37.042631 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":144,"threads_started":1}
I20260810 22:38:37.043946 9539 raft_consensus.cc:374] T 00000000000000000000000000000000 P fbd4515f6a7d4285ae67b0841e4def9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:38:37.044000 9539 raft_consensus.cc:729] T 00000000000000000000000000000000 P fbd4515f6a7d4285ae67b0841e4def9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbd4515f6a7d4285ae67b0841e4def9f, State: Initialized, Role: FOLLOWER
I20260810 22:38:37.044545 9539 consensus_queue.cc:260] T 00000000000000000000000000000000 P fbd4515f6a7d4285ae67b0841e4def9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbd4515f6a7d4285ae67b0841e4def9f" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 41843 } }
I20260810 22:38:37.044894 9539 raft_consensus.cc:388] T 00000000000000000000000000000000 P fbd4515f6a7d4285ae67b0841e4def9f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:38:37.044992 9539 raft_consensus.cc:482] T 00000000000000000000000000000000 P fbd4515f6a7d4285ae67b0841e4def9f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:38:37.045110 9539 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fbd4515f6a7d4285ae67b0841e4def9f [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:38:37.046470 9539 raft_consensus.cc:504] T 00000000000000000000000000000000 P fbd4515f6a7d4285ae67b0841e4def9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbd4515f6a7d4285ae67b0841e4def9f" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 41843 } }
I20260810 22:38:37.046891 9539 leader_election.cc:302] T 00000000000000000000000000000000 P fbd4515f6a7d4285ae67b0841e4def9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fbd4515f6a7d4285ae67b0841e4def9f; no voters:
I20260810 22:38:37.047370 9539 leader_election.cc:288] T 00000000000000000000000000000000 P fbd4515f6a7d4285ae67b0841e4def9f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:38:37.047466 9544 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fbd4515f6a7d4285ae67b0841e4def9f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:38:37.047712 9544 raft_consensus.cc:686] T 00000000000000000000000000000000 P fbd4515f6a7d4285ae67b0841e4def9f [term 1 LEADER]: Becoming Leader. State: Replica: fbd4515f6a7d4285ae67b0841e4def9f, State: Running, Role: LEADER
I20260810 22:38:37.047851 17317 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 22:38:37.048056 9544 consensus_queue.cc:237] T 00000000000000000000000000000000 P fbd4515f6a7d4285ae67b0841e4def9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbd4515f6a7d4285ae67b0841e4def9f" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 41843 } }
I20260810 22:38:37.048982 9539 sys_catalog.cc:564] T 00000000000000000000000000000000 P fbd4515f6a7d4285ae67b0841e4def9f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:38:37.050122 9545 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbd4515f6a7d4285ae67b0841e4def9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader fbd4515f6a7d4285ae67b0841e4def9f. Latest consensus state: current_term: 1 leader_uuid: "fbd4515f6a7d4285ae67b0841e4def9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbd4515f6a7d4285ae67b0841e4def9f" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 41843 } } }
I20260810 22:38:37.050307 9545 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbd4515f6a7d4285ae67b0841e4def9f [sys.catalog]: This master's current role is: LEADER
I20260810 22:38:37.050674 9546 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbd4515f6a7d4285ae67b0841e4def9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fbd4515f6a7d4285ae67b0841e4def9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbd4515f6a7d4285ae67b0841e4def9f" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 41843 } } }
I20260810 22:38:37.050801 9546 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbd4515f6a7d4285ae67b0841e4def9f [sys.catalog]: This master's current role is: LEADER
W20260810 22:38:37.054621 9564 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fbd4515f6a7d4285ae67b0841e4def9f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 22:38:37.054730 9564 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 22:38:37.054809 9557 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:38:37.055784 9557 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:38:37.062068 9542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:37.050866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57953 (local address 127.16.233.126:41843)
0810 22:38:37.050983 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:37.050988 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:37.052226 (+ 1238us) server_negotiation.cc:407] Connection header received
0810 22:38:37.052402 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:37.052406 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:37.052462 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:37.052572 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:37.053712 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:37.054316 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:37.055433 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:37.055724 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:37.055875 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:37.057623 (+ 1748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:37.057628 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:37.057631 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:38:37.057666 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:37.059696 (+ 2030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:37.060305 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:37.060309 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:37.060311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:37.060421 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:37.060801 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:37.060858 (+ 57us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:37.060863 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:37.061153 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:37.061323 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:37.061705 (+ 382us) server_negotiation.cc:299] Negotiation successful
0810 22:38:37.061856 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 22:38:37.063298 9557 catalog_manager.cc:1347] Generated new cluster ID: a333bd9cfadf4abcbe9a5d6cddd799d8
I20260810 22:38:37.063373 9557 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:38:37.086814 9557 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:38:37.087925 9557 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:38:37.100318 9557 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fbd4515f6a7d4285ae67b0841e4def9f: Generated new TSK 0
I20260810 22:38:37.101359 9557 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:38:37.112353 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 9464
2026-08-10T22:38:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1551 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786401215402979-17317-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:38:37 dist-test-slave-77v9 krb5kdc[9570](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:38:37 dist-test-slave-77v9 krb5kdc[9570](info): set up 2 sockets
Aug 10 22:38:37 dist-test-slave-77v9 krb5kdc[9570](info): commencing operation
krb5kdc: starting...
W20260810 22:38:38.309321 17317 mini_kdc.cc:121] Time spent starting KDC: real 1.171s user 0.006s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Aug 10 22:38:38 dist-test-slave-77v9 krb5kdc[9570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:38:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:38:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.233.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab.
Entry for principal kudu/127.16.233.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab.
I20260810 22:38:38.457436 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786401215402979-17317-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786401215402979-17317-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786401215402979-17317-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:41045
--webserver_interface=127.16.233.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab
--principal=kudu/127.16.233.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:40235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.126:41045
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:38:38.577244 9587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:38:38.577663 9587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:38:38.577764 9587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:38:38.582129 9587 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:38:38.582247 9587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:38:38.582283 9587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:38:38.582340 9587 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:38:38.582417 9587 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:38:38.588555 9587 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:40235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786401215402979-17317-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786401215402979-17317-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.126:41045
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab
--principal=kudu/127.16.233.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:41045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786401215402979-17317-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:38:38.589903 9587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:38:38.591029 9587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:38:38.601284 9593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:38.601287 9595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:38.601287 9592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:38:38.601361 9587 server_base.cc:1034] running on GCE node
I20260810 22:38:38.602232 9587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:38:38.603696 9587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:38:38.604959 9587 hybrid_clock.cc:648] HybridClock initialized: now 1786401518604937 us; error 35 us; skew 500 ppm
Aug 10 22:38:38 dist-test-slave-77v9 krb5kdc[9570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401518, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:38:38.608501 9587 init.cc:377] Logged in from keytab as kudu/127.16.233.126@KRBTEST.COM (short username kudu)
I20260810 22:38:38.609735 9587 webserver.cc:460] Webserver started at http://127.16.233.126:35605/ using document root <none> and password file <none>
I20260810 22:38:38.610371 9587 fs_manager.cc:359] Metadata directory not provided
I20260810 22:38:38.610445 9587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:38:38.610687 9587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:38:38.613171 9587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786401215402979-17317-0/minicluster-data/master-0/data/instance:
uuid: "ff837218a121481abd829dd47f0816c9"
format_stamp: "Formatted at 2026-08-10 22:38:38 on dist-test-slave-77v9"
server_key: "1570f11c52b83af6d6e1a0cd91de0e94"
server_key_iv: "9446e95c25f610afd99e4ca4ef6bdd76"
server_key_version: "encryptionkey@0"
I20260810 22:38:38.613847 9587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786401215402979-17317-0/minicluster-data/master-0/wal/instance:
uuid: "ff837218a121481abd829dd47f0816c9"
format_stamp: "Formatted at 2026-08-10 22:38:38 on dist-test-slave-77v9"
server_key: "1570f11c52b83af6d6e1a0cd91de0e94"
server_key_iv: "9446e95c25f610afd99e4ca4ef6bdd76"
server_key_version: "encryptionkey@0"
I20260810 22:38:38.618458 9587 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 22:38:38.621618 9602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:38.623126 9587 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 22:38:38.623317 9587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786401215402979-17317-0/minicluster-data/master-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786401215402979-17317-0/minicluster-data/master-0/wal
uuid: "ff837218a121481abd829dd47f0816c9"
format_stamp: "Formatted at 2026-08-10 22:38:38 on dist-test-slave-77v9"
server_key: "1570f11c52b83af6d6e1a0cd91de0e94"
server_key_iv: "9446e95c25f610afd99e4ca4ef6bdd76"
server_key_version: "encryptionkey@0"
I20260810 22:38:38.623457 9587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786401215402979-17317-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786401215402979-17317-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786401215402979-17317-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:38:38.639117 9587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:38:38.642689 9587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:38:38.642931 9587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:38:38.651368 9587 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.126:41045
I20260810 22:38:38.651465 9662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.126:41045 every 8 connection(s)
I20260810 22:38:38.652508 9587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
I20260810 22:38:38.655869 9663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:38:38.656478 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 9587
I20260810 22:38:38.656654 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786401215402979-17317-0/minicluster-data/master-0/wal/instance
I20260810 22:38:38.657099 17317 external_mini_cluster.cc:1428] Setting key 3f5adb36789210dcfccb8ae7bbf424be
I20260810 22:38:38.664067 9663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff837218a121481abd829dd47f0816c9: Bootstrap starting.
I20260810 22:38:38.667460 9663 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff837218a121481abd829dd47f0816c9: Neither blocks nor log segments found. Creating new log.
I20260810 22:38:38.668274 9666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:38.658728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46169 (local address 127.16.233.126:41045)
0810 22:38:38.659350 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:38.659368 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:38.659410 (+ 42us) server_negotiation.cc:407] Connection header received
0810 22:38:38.660205 (+ 795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:38.660227 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:38.660638 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:38.661105 (+ 467us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:38.661591 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:38.662702 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:38.663763 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:38.664399 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:38.664609 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:38.665877 (+ 1268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:38.665885 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:38.665904 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:38:38.666544 (+ 640us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:38.666633 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:38:38.666668 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:38.666775 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:38.666970 (+ 195us) server_negotiation.cc:299] Negotiation successful
0810 22:38:38.667356 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":180,"threads_started":1}
I20260810 22:38:38.668377 9663 log.cc:824] T 00000000000000000000000000000000 P ff837218a121481abd829dd47f0816c9: Log is configured to *not* fsync() on all Append() calls
I20260810 22:38:38.671047 9663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff837218a121481abd829dd47f0816c9: No bootstrap required, opened a new log
I20260810 22:38:38.671803 17317 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 22:38:38.676577 9663 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff837218a121481abd829dd47f0816c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff837218a121481abd829dd47f0816c9" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 41045 } }
I20260810 22:38:38.676934 9663 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff837218a121481abd829dd47f0816c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:38:38.677034 9663 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff837218a121481abd829dd47f0816c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff837218a121481abd829dd47f0816c9, State: Initialized, Role: FOLLOWER
I20260810 22:38:38.677642 9663 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff837218a121481abd829dd47f0816c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff837218a121481abd829dd47f0816c9" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 41045 } }
I20260810 22:38:38.677862 9663 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff837218a121481abd829dd47f0816c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:38:38.678006 9663 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff837218a121481abd829dd47f0816c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:38:38.678146 9663 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff837218a121481abd829dd47f0816c9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:38:38.679452 9666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:38.674460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55533 (local address 127.16.233.126:41045)
0810 22:38:38.674628 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:38.674635 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:38.674790 (+ 155us) server_negotiation.cc:407] Connection header received
0810 22:38:38.674889 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:38.674894 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:38.674965 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:38.675077 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:38.675347 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:38.676206 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:38.677332 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:38.677719 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:38.677859 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:38.678979 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:38.678983 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:38.678986 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:38:38.679003 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:38.679041 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:38:38.679055 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:38.679133 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:38.679216 (+ 83us) server_negotiation.cc:299] Negotiation successful
0810 22:38:38.679285 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 22:38:38.679698 9663 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff837218a121481abd829dd47f0816c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff837218a121481abd829dd47f0816c9" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 41045 } }
I20260810 22:38:38.680114 9663 leader_election.cc:302] T 00000000000000000000000000000000 P ff837218a121481abd829dd47f0816c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff837218a121481abd829dd47f0816c9; no voters:
I20260810 22:38:38.680545 9663 leader_election.cc:288] T 00000000000000000000000000000000 P ff837218a121481abd829dd47f0816c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:38:38.680655 9673 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff837218a121481abd829dd47f0816c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:38:38.681126 9673 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff837218a121481abd829dd47f0816c9 [term 1 LEADER]: Becoming Leader. State: Replica: ff837218a121481abd829dd47f0816c9, State: Running, Role: LEADER
I20260810 22:38:38.681506 9673 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff837218a121481abd829dd47f0816c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff837218a121481abd829dd47f0816c9" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 41045 } }
I20260810 22:38:38.682098 9663 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff837218a121481abd829dd47f0816c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:38:38.683557 9674 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff837218a121481abd829dd47f0816c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff837218a121481abd829dd47f0816c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff837218a121481abd829dd47f0816c9" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 41045 } } }
I20260810 22:38:38.683568 9675 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff837218a121481abd829dd47f0816c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff837218a121481abd829dd47f0816c9. Latest consensus state: current_term: 1 leader_uuid: "ff837218a121481abd829dd47f0816c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff837218a121481abd829dd47f0816c9" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 41045 } } }
I20260810 22:38:38.683753 9675 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff837218a121481abd829dd47f0816c9 [sys.catalog]: This master's current role is: LEADER
I20260810 22:38:38.683753 9674 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff837218a121481abd829dd47f0816c9 [sys.catalog]: This master's current role is: LEADER
I20260810 22:38:38.684866 9683 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:38:38.688405 9683 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:38:38.695060 9683 catalog_manager.cc:1347] Generated new cluster ID: d33428f429314d7d8a957a56678afdec
I20260810 22:38:38.695160 9683 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:38:38.703291 9683 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:38:38.704507 9683 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:38:38.714402 9683 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff837218a121481abd829dd47f0816c9: Generated new TSK 0
I20260810 22:38:38.715444 9683 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:38:38.729446 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 9587
2026-08-10T22:38:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1615 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786401215402979-17317-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:38:38 dist-test-slave-77v9 krb5kdc[9696](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:38:38 dist-test-slave-77v9 krb5kdc[9696](info): set up 2 sockets
Aug 10 22:38:38 dist-test-slave-77v9 krb5kdc[9696](info): commencing operation
krb5kdc: starting...
W20260810 22:38:40.115635 17317 mini_kdc.cc:121] Time spent starting KDC: real 1.362s user 0.009s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Aug 10 22:38:40 dist-test-slave-77v9 krb5kdc[9696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:38:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:38:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.233.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab.
Entry for principal kudu/127.16.233.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab.
I20260810 22:38:40.333772 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786401215402979-17317-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786401215402979-17317-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786401215402979-17317-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:33881
--webserver_interface=127.16.233.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab
--principal=kudu/127.16.233.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:35635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.126:33881
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:38:40.641217 9709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:38:40.641948 9709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:38:40.642163 9709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:38:40.650734 9709 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:38:40.651149 9709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:38:40.651348 9709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:38:40.651451 9709 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:38:40.651535 9709 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:38:40.663110 9709 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:35635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786401215402979-17317-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786401215402979-17317-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.126:33881
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab
--principal=kudu/127.16.233.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:33881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786401215402979-17317-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:38:40.666258 9709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:38:40.669140 9709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:38:40.690034 9715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:40.690016 9716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:40.690016 9718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:38:40.691730 9709 server_base.cc:1034] running on GCE node
I20260810 22:38:40.693279 9709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:38:40.696002 9709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:38:40.699648 9709 hybrid_clock.cc:648] HybridClock initialized: now 1786401520699528 us; error 107 us; skew 500 ppm
Aug 10 22:38:40 dist-test-slave-77v9 krb5kdc[9696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401520, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:38:40.712795 9709 init.cc:377] Logged in from keytab as kudu/127.16.233.126@KRBTEST.COM (short username kudu)
I20260810 22:38:40.715348 9709 webserver.cc:460] Webserver started at http://127.16.233.126:44861/ using document root <none> and password file <none>
I20260810 22:38:40.716753 9709 fs_manager.cc:359] Metadata directory not provided
I20260810 22:38:40.717049 9709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:38:40.717548 9709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:38:40.722955 9709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786401215402979-17317-0/minicluster-data/master-0/data/instance:
uuid: "6df334141031493683f9e5d77ea16740"
format_stamp: "Formatted at 2026-08-10 22:38:40 on dist-test-slave-77v9"
server_key: "08752583490a11bc114d2799021e078c"
server_key_iv: "993a25c3c404a19967aeed1ba710aada"
server_key_version: "encryptionkey@0"
I20260810 22:38:40.724499 9709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786401215402979-17317-0/minicluster-data/master-0/wal/instance:
uuid: "6df334141031493683f9e5d77ea16740"
format_stamp: "Formatted at 2026-08-10 22:38:40 on dist-test-slave-77v9"
server_key: "08752583490a11bc114d2799021e078c"
server_key_iv: "993a25c3c404a19967aeed1ba710aada"
server_key_version: "encryptionkey@0"
I20260810 22:38:40.737084 9709 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.011s sys 0.000s
I20260810 22:38:40.743984 9727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:40.747602 9709 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.000s
I20260810 22:38:40.748010 9709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786401215402979-17317-0/minicluster-data/master-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786401215402979-17317-0/minicluster-data/master-0/wal
uuid: "6df334141031493683f9e5d77ea16740"
format_stamp: "Formatted at 2026-08-10 22:38:40 on dist-test-slave-77v9"
server_key: "08752583490a11bc114d2799021e078c"
server_key_iv: "993a25c3c404a19967aeed1ba710aada"
server_key_version: "encryptionkey@0"
I20260810 22:38:40.748312 9709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786401215402979-17317-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786401215402979-17317-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786401215402979-17317-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:38:40.761651 9709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:38:40.770896 9709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:38:40.771386 9709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:38:40.788398 9709 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.126:33881
I20260810 22:38:40.788555 9783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.126:33881 every 8 connection(s)
I20260810 22:38:40.790462 9709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
I20260810 22:38:40.797458 9784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:38:40.798888 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 9709
I20260810 22:38:40.799220 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786401215402979-17317-0/minicluster-data/master-0/wal/instance
I20260810 22:38:40.800045 17317 external_mini_cluster.cc:1428] Setting key 225f0fa963203b963b670db328342da6
I20260810 22:38:40.811468 9787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:40.803530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41247 (local address 127.16.233.126:33881)
0810 22:38:40.804678 (+ 1148us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:40.804702 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:40.804869 (+ 167us) server_negotiation.cc:407] Connection header received
0810 22:38:40.806165 (+ 1296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:40.806201 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:40.806807 (+ 606us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:40.807506 (+ 699us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:40.809267 (+ 1761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:40.809279 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:40.809304 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:38:40.809324 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:40.809420 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:38:40.809459 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:40.809744 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:40.809884 (+ 140us) server_negotiation.cc:299] Negotiation successful
0810 22:38:40.810199 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":759,"thread_start_us":383,"threads_started":1}
I20260810 22:38:40.813360 9784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6df334141031493683f9e5d77ea16740: Bootstrap starting.
I20260810 22:38:40.816591 17317 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 22:38:40.818216 9784 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6df334141031493683f9e5d77ea16740: Neither blocks nor log segments found. Creating new log.
I20260810 22:38:40.819940 9784 log.cc:824] T 00000000000000000000000000000000 P 6df334141031493683f9e5d77ea16740: Log is configured to *not* fsync() on all Append() calls
I20260810 22:38:40.825067 9787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:40.820560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56893 (local address 127.16.233.126:33881)
0810 22:38:40.820855 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:40.820862 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:40.821157 (+ 295us) server_negotiation.cc:407] Connection header received
0810 22:38:40.822268 (+ 1111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:40.822273 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:40.822386 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:40.822522 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:40.824013 (+ 1491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:40.824016 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:40.824018 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:38:40.824041 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:40.824083 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:38:40.824102 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:40.824242 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:40.824465 (+ 223us) server_negotiation.cc:299] Negotiation successful
0810 22:38:40.824551 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 22:38:40.825990 9784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6df334141031493683f9e5d77ea16740: No bootstrap required, opened a new log
I20260810 22:38:40.833962 9784 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6df334141031493683f9e5d77ea16740 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6df334141031493683f9e5d77ea16740" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 33881 } }
I20260810 22:38:40.834525 9784 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6df334141031493683f9e5d77ea16740 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:38:40.834609 9784 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6df334141031493683f9e5d77ea16740 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6df334141031493683f9e5d77ea16740, State: Initialized, Role: FOLLOWER
I20260810 22:38:40.835294 9784 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6df334141031493683f9e5d77ea16740 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6df334141031493683f9e5d77ea16740" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 33881 } }
I20260810 22:38:40.835594 9784 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6df334141031493683f9e5d77ea16740 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:38:40.835814 9784 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6df334141031493683f9e5d77ea16740 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:38:40.836023 9784 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6df334141031493683f9e5d77ea16740 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:38:40.838152 9784 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6df334141031493683f9e5d77ea16740 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6df334141031493683f9e5d77ea16740" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 33881 } }
I20260810 22:38:40.838856 9784 leader_election.cc:302] T 00000000000000000000000000000000 P 6df334141031493683f9e5d77ea16740 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6df334141031493683f9e5d77ea16740; no voters:
I20260810 22:38:40.839491 9784 leader_election.cc:288] T 00000000000000000000000000000000 P 6df334141031493683f9e5d77ea16740 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:38:40.839695 9797 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6df334141031493683f9e5d77ea16740 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:38:40.840189 9797 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6df334141031493683f9e5d77ea16740 [term 1 LEADER]: Becoming Leader. State: Replica: 6df334141031493683f9e5d77ea16740, State: Running, Role: LEADER
I20260810 22:38:40.840780 9797 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6df334141031493683f9e5d77ea16740 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6df334141031493683f9e5d77ea16740" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 33881 } }
I20260810 22:38:40.842033 9784 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6df334141031493683f9e5d77ea16740 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:38:40.844637 9799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6df334141031493683f9e5d77ea16740 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6df334141031493683f9e5d77ea16740. Latest consensus state: current_term: 1 leader_uuid: "6df334141031493683f9e5d77ea16740" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6df334141031493683f9e5d77ea16740" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 33881 } } }
I20260810 22:38:40.844708 9798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6df334141031493683f9e5d77ea16740 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6df334141031493683f9e5d77ea16740" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6df334141031493683f9e5d77ea16740" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 33881 } } }
I20260810 22:38:40.845079 9799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6df334141031493683f9e5d77ea16740 [sys.catalog]: This master's current role is: LEADER
I20260810 22:38:40.845117 9798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6df334141031493683f9e5d77ea16740 [sys.catalog]: This master's current role is: LEADER
I20260810 22:38:40.846037 9806 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:38:40.853154 9806 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:38:40.867897 9806 catalog_manager.cc:1347] Generated new cluster ID: 356345b0765841eaa2aed0848c397ac4
I20260810 22:38:40.868057 9806 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:38:40.894515 9806 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:38:40.895372 9806 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:38:40.912638 9806 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6df334141031493683f9e5d77ea16740: Generated new TSK 0
I20260810 22:38:40.914507 9806 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:38:40.933774 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 9709
2026-08-10T22:38:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2216 ms)
[----------] 18 tests from AuthTokenIssuingTest (29007 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:38:41 dist-test-slave-77v9 krb5kdc[9819](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:38:41 dist-test-slave-77v9 krb5kdc[9819](info): set up 2 sockets
Aug 10 22:38:41 dist-test-slave-77v9 krb5kdc[9819](info): commencing operation
krb5kdc: starting...
W20260810 22:38:42.286175 17317 mini_kdc.cc:121] Time spent starting KDC: real 1.315s user 0.008s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Aug 10 22:38:42 dist-test-slave-77v9 krb5kdc[9819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:38:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:38:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.233.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab.
Entry for principal kudu/127.16.233.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab.
I20260810 22:38:42.442052 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:37779
--webserver_interface=127.16.233.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab
--principal=kudu/127.16.233.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:35705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.126:37779,127.16.233.125:33363,127.16.233.124:35937
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:38:42.577337 9833 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 22:38:42.577783 9833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:38:42.577971 9833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:38:42.578061 9833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:38:42.582772 9833 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:38:42.582904 9833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:38:42.582942 9833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:38:42.582988 9833 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:38:42.583020 9833 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:38:42.588925 9833 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:35705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.126:37779,127.16.233.125:33363,127.16.233.124:35937
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab
--principal=kudu/127.16.233.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:37779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9833
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:38:42.590282 9833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:38:42.591480 9833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:38:42.601219 9842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:42.601229 9839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:42.601228 9840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:38:42.601615 9833 server_base.cc:1034] running on GCE node
I20260810 22:38:42.602386 9833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:38:42.603766 9833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:38:42.604991 9833 hybrid_clock.cc:648] HybridClock initialized: now 1786401522604956 us; error 52 us; skew 500 ppm
Aug 10 22:38:42 dist-test-slave-77v9 krb5kdc[9819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401522, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:38:42.608328 9833 init.cc:377] Logged in from keytab as kudu/127.16.233.126@KRBTEST.COM (short username kudu)
I20260810 22:38:42.609594 9833 webserver.cc:460] Webserver started at http://127.16.233.126:33029/ using document root <none> and password file <none>
I20260810 22:38:42.610248 9833 fs_manager.cc:359] Metadata directory not provided
I20260810 22:38:42.610359 9833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:38:42.610683 9833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:38:42.613129 9833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/minicluster-data/master-0/data/instance:
uuid: "ecf84eda480f48ec8903315ccf9f81d9"
format_stamp: "Formatted at 2026-08-10 22:38:42 on dist-test-slave-77v9"
server_key: "a98bc47ddc99437dfb53a4c1e111d7e1"
server_key_iv: "b773e13ae6330fd818f54ab289092884"
server_key_version: "encryptionkey@0"
I20260810 22:38:42.613798 9833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/minicluster-data/master-0/wal/instance:
uuid: "ecf84eda480f48ec8903315ccf9f81d9"
format_stamp: "Formatted at 2026-08-10 22:38:42 on dist-test-slave-77v9"
server_key: "a98bc47ddc99437dfb53a4c1e111d7e1"
server_key_iv: "b773e13ae6330fd818f54ab289092884"
server_key_version: "encryptionkey@0"
I20260810 22:38:42.618278 9833 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 22:38:42.621416 9849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:42.622991 9833 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 22:38:42.623190 9833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/minicluster-data/master-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/minicluster-data/master-0/wal
uuid: "ecf84eda480f48ec8903315ccf9f81d9"
format_stamp: "Formatted at 2026-08-10 22:38:42 on dist-test-slave-77v9"
server_key: "a98bc47ddc99437dfb53a4c1e111d7e1"
server_key_iv: "b773e13ae6330fd818f54ab289092884"
server_key_version: "encryptionkey@0"
I20260810 22:38:42.623345 9833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:38:42.643196 9833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:38:42.647710 9833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:38:42.648003 9833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:38:42.657612 9833 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.126:37779
I20260810 22:38:42.658020 9906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.126:37779 every 8 connection(s)
I20260810 22:38:42.658857 9833 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
I20260810 22:38:42.662451 9907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:38:42.662559 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 9833
I20260810 22:38:42.662731 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/minicluster-data/master-0/wal/instance
I20260810 22:38:42.663143 17317 external_mini_cluster.cc:1428] Setting key 83a1ee57f6b36957d1798eebcb3bfdcb
I20260810 22:38:42.666914 9907 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.233.126" port: 37779 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.16.233.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.125@KRBTEST.COM" created.
Aug 10 22:38:42 dist-test-slave-77v9 krb5kdc[9819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401522, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.126@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:38:42.685303 9911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:42.671025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46096 (local address 127.16.233.126:37779)
0810 22:38:42.671413 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:42.671427 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:42.671483 (+ 56us) server_negotiation.cc:407] Connection header received
0810 22:38:42.672088 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:42.672096 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:42.672393 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:42.672622 (+ 229us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:42.673647 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:42.675183 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:42.676364 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:42.676789 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:42.677076 (+ 287us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:42.679926 (+ 2850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:42.679932 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:42.679944 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:38:42.679967 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:42.681791 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:42.682356 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:42.682363 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:42.682364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:42.682483 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:42.682819 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:42.682822 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:42.682824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:42.683266 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:42.683417 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:42.683819 (+ 402us) server_negotiation.cc:299] Negotiation successful
0810 22:38:42.684098 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":98,"threads_started":1}
I20260810 22:38:42.685303 9910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:42.670009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:37779 (local address 127.0.0.1:46096)
0810 22:38:42.670568 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:42.670626 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 22:38:42.671703 (+ 1077us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:38:42.672626 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:38:42.672631 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:38:42.673156 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:38:42.673434 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:42.673448 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:42.675343 (+ 1895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:42.675347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:42.676219 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:42.676230 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:42.677066 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:42.677070 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:42.677220 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:42.677961 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:38:42.677981 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:38:42.679727 (+ 1746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:38:42.681962 (+ 2235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:38:42.681973 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:38:42.681986 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:38:42.682235 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:38:42.682587 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:38:42.682592 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:38:42.682594 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:38:42.682717 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:38:42.683394 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:38:42.683401 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:38:42.683558 (+ 157us) client_negotiation.cc:769] Sending connection context
0810 22:38:42.683735 (+ 177us) client_negotiation.cc:240] Negotiation successful
0810 22:38:42.684182 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":114,"thread_start_us":172,"threads_started":1}
I20260810 22:38:42.688613 9907 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.233.125" port: 33363 } has no permanent_uuid. Determining permanent_uuid...
I20260810 22:38:42.689527 9910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:42.689210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.125:33363 (local address 127.0.0.1:40970)
0810 22:38:42.689347 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:42.689452 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.125:33363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260810 22:38:42.689817 9852 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.233.125:33363: connect: Connection refused (error 111)
I20260810 22:38:42.690852 9910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:42.690636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.125:33363 (local address 127.0.0.1:40980)
0810 22:38:42.690744 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:42.690791 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.125:33363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 22:38:42.691040 9907 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.233.125:33363: Network error: Client connection negotiation failed: client connection to 127.16.233.125:33363: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.125.keytab.
Entry for principal kudu/127.16.233.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.125.keytab.
I20260810 22:38:42.697199 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.125:33363
--webserver_interface=127.16.233.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.125.keytab
--principal=kudu/127.16.233.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:35705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.126:37779,127.16.233.125:33363,127.16.233.124:35937
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 22:38:42.740398 9907 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.233.125" port: 33363 } attempt: 1
I20260810 22:38:42.741636 9910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:42.741293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.125:33363 (local address 127.0.0.1:40988)
0810 22:38:42.741470 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:42.741548 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.125:33363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260810 22:38:42.742479 9910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:42.742237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.125:33363 (local address 127.0.0.1:41002)
0810 22:38:42.742358 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:42.742414 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.125:33363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260810 22:38:42.742627 9907 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.233.125:33363: Network error: Client connection negotiation failed: client connection to 127.16.233.125:33363: connect: Connection refused (error 111)
I20260810 22:38:42.810825 9907 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.233.125" port: 33363 } attempt: 2
I20260810 22:38:42.811815 9910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:42.811490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.125:33363 (local address 127.0.0.1:41004)
0810 22:38:42.811662 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:42.811734 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.125:33363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260810 22:38:42.812572 9910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:42.812353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.125:33363 (local address 127.0.0.1:41006)
0810 22:38:42.812462 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:42.812513 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.125:33363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260810 22:38:42.812709 9907 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.233.125:33363: Network error: Client connection negotiation failed: client connection to 127.16.233.125:33363: connect: Connection refused (error 111)
W20260810 22:38:42.828621 9915 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 22:38:42.829046 9915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:38:42.829109 9915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:38:42.829195 9915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:38:42.833632 9915 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:38:42.833725 9915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:38:42.833748 9915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:38:42.833767 9915 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:38:42.833783 9915 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:38:42.839723 9915 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:35705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.126:37779,127.16.233.125:33363,127.16.233.124:35937
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.125.keytab
--principal=kudu/127.16.233.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.125:33363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.233.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9915
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:38:42.841046 9915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:38:42.842154 9915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:38:42.851727 9921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:42.851727 9924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:42.851755 9922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:38:42.852211 9915 server_base.cc:1034] running on GCE node
I20260810 22:38:42.852936 9915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:38:42.854249 9915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:38:42.855443 9915 hybrid_clock.cc:648] HybridClock initialized: now 1786401522855422 us; error 39 us; skew 500 ppm
Aug 10 22:38:42 dist-test-slave-77v9 krb5kdc[9819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401522, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:38:42.858776 9915 init.cc:377] Logged in from keytab as kudu/127.16.233.125@KRBTEST.COM (short username kudu)
I20260810 22:38:42.859831 9915 webserver.cc:460] Webserver started at http://127.16.233.125:42237/ using document root <none> and password file <none>
I20260810 22:38:42.860518 9915 fs_manager.cc:359] Metadata directory not provided
I20260810 22:38:42.860589 9915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:38:42.860898 9915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:38:42.863214 9915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/minicluster-data/master-1/data/instance:
uuid: "ae8665eb53d548c8ac38052cdf949b1e"
format_stamp: "Formatted at 2026-08-10 22:38:42 on dist-test-slave-77v9"
server_key: "1b494125cb2770037a94b0b27840bd80"
server_key_iv: "78e4b098b2f34d5e72eec3eadb0865b7"
server_key_version: "encryptionkey@0"
I20260810 22:38:42.863878 9915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/minicluster-data/master-1/wal/instance:
uuid: "ae8665eb53d548c8ac38052cdf949b1e"
format_stamp: "Formatted at 2026-08-10 22:38:42 on dist-test-slave-77v9"
server_key: "1b494125cb2770037a94b0b27840bd80"
server_key_iv: "78e4b098b2f34d5e72eec3eadb0865b7"
server_key_version: "encryptionkey@0"
I20260810 22:38:42.868497 9915 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260810 22:38:42.871420 9932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:42.872990 9915 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:38:42.873150 9915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/minicluster-data/master-1/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/minicluster-data/master-1/wal
uuid: "ae8665eb53d548c8ac38052cdf949b1e"
format_stamp: "Formatted at 2026-08-10 22:38:42 on dist-test-slave-77v9"
server_key: "1b494125cb2770037a94b0b27840bd80"
server_key_iv: "78e4b098b2f34d5e72eec3eadb0865b7"
server_key_version: "encryptionkey@0"
I20260810 22:38:42.873279 9915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:38:42.903913 9907 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.233.125" port: 33363 } attempt: 3
I20260810 22:38:42.905004 9910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:42.904535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.125:33363 (local address 127.0.0.1:41018)
0810 22:38:42.904749 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:42.904910 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.125:33363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260810 22:38:42.905870 9910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:42.905584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.125:33363 (local address 127.0.0.1:41034)
0810 22:38:42.905730 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:42.905802 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.125:33363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260810 22:38:42.906054 9907 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.233.125:33363: Network error: Client connection negotiation failed: client connection to 127.16.233.125:33363: connect: Connection refused (error 111)
I20260810 22:38:42.918999 9915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:38:42.923296 9915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:38:42.923491 9915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:38:42.932658 9915 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.125:33363
I20260810 22:38:42.933014 9990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.125:33363 every 8 connection(s)
I20260810 22:38:42.933939 9915 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/minicluster-data/master-1/data/info.pb
I20260810 22:38:42.937170 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 9915
I20260810 22:38:42.937294 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/minicluster-data/master-1/wal/instance
I20260810 22:38:42.937427 9991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:38:42.937711 17317 external_mini_cluster.cc:1428] Setting key 31636b0fe10d5a2950be9a98526a97aa
I20260810 22:38:42.941952 9991 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.233.126" port: 37779 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.16.233.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.124@KRBTEST.COM" created.
Aug 10 22:38:42 dist-test-slave-77v9 krb5kdc[9819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401522, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.125@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:38:42.958703 9911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:42.944348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46102 (local address 127.16.233.126:37779)
0810 22:38:42.944688 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:42.944693 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:42.945691 (+ 998us) server_negotiation.cc:407] Connection header received
0810 22:38:42.946799 (+ 1108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:42.946803 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:42.946888 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:42.947012 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:42.948445 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:42.949144 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:42.950559 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:42.950801 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:42.950900 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:42.954129 (+ 3229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:42.954133 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:42.954136 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:38:42.954167 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:42.956013 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:42.956719 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:42.956723 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:42.956725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:42.956890 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:42.957318 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:42.957323 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:42.957326 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:42.957556 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:42.957684 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:42.958333 (+ 649us) server_negotiation.cc:299] Negotiation successful
0810 22:38:42.958502 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169}
I20260810 22:38:42.959685 9994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:42.944861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:37779 (local address 127.0.0.1:46102)
0810 22:38:42.945478 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:42.945539 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 22:38:42.946555 (+ 1016us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:38:42.947258 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:38:42.947266 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:38:42.947726 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:38:42.948273 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:42.948288 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:42.949319 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:42.949322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:42.950437 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:42.950449 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:42.951009 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:42.951013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:42.951187 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:42.952094 (+ 907us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:38:42.952114 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:38:42.953892 (+ 1778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:38:42.956178 (+ 2286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:38:42.956186 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:38:42.956207 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:38:42.956577 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:38:42.956998 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:38:42.957002 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:38:42.957005 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:38:42.957194 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:38:42.957686 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:38:42.957698 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:38:42.958035 (+ 337us) client_negotiation.cc:769] Sending connection context
0810 22:38:42.958264 (+ 229us) client_negotiation.cc:240] Negotiation successful
0810 22:38:42.958682 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":183,"threads_started":1}
I20260810 22:38:42.961359 9991 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.233.125" port: 33363 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 22:38:42 dist-test-slave-77v9 krb5kdc[9819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401522, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.125@KRBTEST.COM for kudu/127.16.233.125@KRBTEST.COM
I20260810 22:38:42.972437 9994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:42.961914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.125:33363 (local address 127.0.0.1:41042)
0810 22:38:42.962082 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:42.962097 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:38:42.962178 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:38:42.962920 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:38:42.962923 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:38:42.963126 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:38:42.963212 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:42.963218 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:42.964470 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:42.964474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:42.965163 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:42.965172 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:42.965816 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:42.965819 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:42.965906 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:42.966468 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:38:42.966482 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:38:42.967968 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:38:42.970566 (+ 2598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:38:42.970571 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:38:42.970573 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:38:42.970832 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:38:42.971186 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:38:42.971189 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:38:42.971191 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:38:42.971281 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:38:42.971852 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:38:42.971854 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:38:42.971946 (+ 92us) client_negotiation.cc:769] Sending connection context
0810 22:38:42.972066 (+ 120us) client_negotiation.cc:240] Negotiation successful
0810 22:38:42.972222 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260810 22:38:42.972456 9996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:42.962044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41042 (local address 127.16.233.125:33363)
0810 22:38:42.962464 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:42.962473 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:42.962491 (+ 18us) server_negotiation.cc:407] Connection header received
0810 22:38:42.962526 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:42.962531 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:42.962684 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:42.962950 (+ 266us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:42.963331 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:42.964343 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:42.965286 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:42.965684 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:42.965814 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:42.968150 (+ 2336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:42.968164 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:42.968178 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:38:42.968201 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:42.970425 (+ 2224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:42.970961 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:42.970970 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:42.970975 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:42.971082 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:42.971371 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:42.971376 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:42.971378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:42.971740 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:42.971898 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:42.972178 (+ 280us) server_negotiation.cc:299] Negotiation successful
0810 22:38:42.972316 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":215,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.124.keytab.
Entry for principal kudu/127.16.233.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.124.keytab.
I20260810 22:38:42.974239 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.124:35937
--webserver_interface=127.16.233.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.124.keytab
--principal=kudu/127.16.233.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:35705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.126:37779,127.16.233.125:33363,127.16.233.124:35937
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 22:38:42.975126 9991 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.233.124" port: 35937 } has no permanent_uuid. Determining permanent_uuid...
I20260810 22:38:42.976056 9994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:42.975722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.124:35937 (local address 127.0.0.1:42736)
0810 22:38:42.975897 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:42.975979 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.124:35937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260810 22:38:42.976313 9934 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.233.124:35937: connect: Connection refused (error 111)
I20260810 22:38:42.977391 9994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:42.977132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.124:35937 (local address 127.0.0.1:42752)
0810 22:38:42.977288 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:42.977333 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.124:35937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260810 22:38:42.977665 9991 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.233.124:35937: Network error: Client connection negotiation failed: client connection to 127.16.233.124:35937: connect: Connection refused (error 111)
I20260810 22:38:43.027045 9991 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.233.124" port: 35937 } attempt: 1
I20260810 22:38:43.028473 9994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:43.027856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.124:35937 (local address 127.0.0.1:42766)
0810 22:38:43.028221 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:43.028328 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.124:35937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226}
I20260810 22:38:43.029541 9994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:43.029226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.124:35937 (local address 127.0.0.1:42782)
0810 22:38:43.029388 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:43.029456 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.124:35937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260810 22:38:43.029805 9991 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.233.124:35937: Network error: Client connection negotiation failed: client connection to 127.16.233.124:35937: connect: Connection refused (error 111)
I20260810 22:38:43.049356 9907 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.233.125" port: 33363 } attempt: 4
Aug 10 22:38:43 dist-test-slave-77v9 krb5kdc[9819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401522, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.126@KRBTEST.COM for kudu/127.16.233.125@KRBTEST.COM
I20260810 22:38:43.061648 9910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:43.050245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.125:33363 (local address 127.0.0.1:41046)
0810 22:38:43.050478 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:43.050498 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 22:38:43.050618 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:38:43.050978 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:38:43.050982 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:38:43.051183 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:38:43.051317 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:43.051324 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:43.053099 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:43.053103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:43.053742 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:43.053754 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:43.054460 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:43.054463 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:43.054561 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:43.055253 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:38:43.055268 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:38:43.057030 (+ 1762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:38:43.059482 (+ 2452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:38:43.059485 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:38:43.059487 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:38:43.059836 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:38:43.060222 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:38:43.060225 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:38:43.060227 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:38:43.060380 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:38:43.061066 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:38:43.061069 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:38:43.061174 (+ 105us) client_negotiation.cc:769] Sending connection context
0810 22:38:43.061278 (+ 104us) client_negotiation.cc:240] Negotiation successful
0810 22:38:43.061493 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114}
I20260810 22:38:43.061755 9996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:43.050365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41046 (local address 127.16.233.125:33363)
0810 22:38:43.050545 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:43.050549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:43.050570 (+ 21us) server_negotiation.cc:407] Connection header received
0810 22:38:43.050739 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:43.050745 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:43.050812 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:43.050931 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:43.051463 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:43.052876 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:43.053905 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:43.054283 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:43.054411 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:43.057219 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:43.057226 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:43.057230 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:38:43.057267 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:43.059320 (+ 2053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:43.059964 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:43.059968 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:43.059970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:43.060077 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:43.060492 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:43.060496 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:43.060498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:43.060744 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:43.060872 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:43.061414 (+ 542us) server_negotiation.cc:299] Negotiation successful
0810 22:38:43.061592 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260810 22:38:43.063007 9907 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.233.124" port: 35937 } has no permanent_uuid. Determining permanent_uuid...
I20260810 22:38:43.064121 9910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:43.063627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.124:35937 (local address 127.0.0.1:42792)
0810 22:38:43.063939 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:43.064019 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.124:35937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260810 22:38:43.065150 9910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:43.064891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.124:35937 (local address 127.0.0.1:42806)
0810 22:38:43.065026 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:43.065083 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.124:35937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260810 22:38:43.065333 9907 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.233.124:35937: Network error: Client connection negotiation failed: client connection to 127.16.233.124:35937: connect: Connection refused (error 111)
I20260810 22:38:43.098105 9991 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.233.124" port: 35937 } attempt: 2
I20260810 22:38:43.099274 9994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:43.098851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.124:35937 (local address 127.0.0.1:42810)
0810 22:38:43.099071 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:43.099161 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.124:35937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260810 22:38:43.100263 9994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:43.099962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.124:35937 (local address 127.0.0.1:42818)
0810 22:38:43.100117 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:43.100184 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.124:35937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260810 22:38:43.100456 9991 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.233.124:35937: Network error: Client connection negotiation failed: client connection to 127.16.233.124:35937: connect: Connection refused (error 111)
W20260810 22:38:43.115110 9997 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 22:38:43.115499 9997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:38:43.115535 9997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:38:43.115577 9997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:38:43.119956 9997 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:38:43.120059 9997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:38:43.120079 9997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:38:43.120096 9997 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:38:43.120113 9997 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:38:43.124562 9907 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.233.124" port: 35937 } attempt: 1
I20260810 22:38:43.125801 9910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:43.125343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.124:35937 (local address 127.0.0.1:42828)
0810 22:38:43.125549 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:43.125632 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.124:35937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260810 22:38:43.126833 9910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:43.126514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.124:35937 (local address 127.0.0.1:42834)
0810 22:38:43.126686 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:43.126750 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.124:35937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260810 22:38:43.126801 9997 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:35705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.126:37779,127.16.233.125:33363,127.16.233.124:35937
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.124.keytab
--principal=kudu/127.16.233.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.124:35937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.233.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9997
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
W20260810 22:38:43.127095 9907 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.233.124:35937: Network error: Client connection negotiation failed: client connection to 127.16.233.124:35937: connect: Connection refused (error 111)
I20260810 22:38:43.128103 9997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:38:43.129420 9997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:38:43.138993 10005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:43.139036 10004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:43.139036 10007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:38:43.139492 9997 server_base.cc:1034] running on GCE node
I20260810 22:38:43.140210 9997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:38:43.141537 9997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:38:43.142724 9997 hybrid_clock.cc:648] HybridClock initialized: now 1786401523142673 us; error 70 us; skew 500 ppm
Aug 10 22:38:43 dist-test-slave-77v9 krb5kdc[9819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401523, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:38:43.146337 9997 init.cc:377] Logged in from keytab as kudu/127.16.233.124@KRBTEST.COM (short username kudu)
I20260810 22:38:43.147629 9997 webserver.cc:460] Webserver started at http://127.16.233.124:44899/ using document root <none> and password file <none>
I20260810 22:38:43.148304 9997 fs_manager.cc:359] Metadata directory not provided
I20260810 22:38:43.148415 9997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:38:43.148674 9997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:38:43.151202 9997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/minicluster-data/master-2/data/instance:
uuid: "71b0d06fae4b4112aec3d98020bf94f3"
format_stamp: "Formatted at 2026-08-10 22:38:43 on dist-test-slave-77v9"
server_key: "ccefe1b1a85e7e013f4fb59158988490"
server_key_iv: "7af18b009fef0eb2823cfcf604e90f6c"
server_key_version: "encryptionkey@0"
I20260810 22:38:43.151858 9997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/minicluster-data/master-2/wal/instance:
uuid: "71b0d06fae4b4112aec3d98020bf94f3"
format_stamp: "Formatted at 2026-08-10 22:38:43 on dist-test-slave-77v9"
server_key: "ccefe1b1a85e7e013f4fb59158988490"
server_key_iv: "7af18b009fef0eb2823cfcf604e90f6c"
server_key_version: "encryptionkey@0"
I20260810 22:38:43.156345 9997 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 22:38:43.159427 10014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:43.161099 9997 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:38:43.161285 9997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/minicluster-data/master-2/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/minicluster-data/master-2/wal
uuid: "71b0d06fae4b4112aec3d98020bf94f3"
format_stamp: "Formatted at 2026-08-10 22:38:43 on dist-test-slave-77v9"
server_key: "ccefe1b1a85e7e013f4fb59158988490"
server_key_iv: "7af18b009fef0eb2823cfcf604e90f6c"
server_key_version: "encryptionkey@0"
I20260810 22:38:43.161445 9997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:38:43.189354 9997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:38:43.191671 9991 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.233.124" port: 35937 } attempt: 3
I20260810 22:38:43.192606 9994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:43.192282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.124:35937 (local address 127.0.0.1:42840)
0810 22:38:43.192450 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:43.192521 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.124:35937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260810 22:38:43.193626 9994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:43.193336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.124:35937 (local address 127.0.0.1:42844)
0810 22:38:43.193505 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:43.193561 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.124:35937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260810 22:38:43.193773 9991 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.233.124:35937: Network error: Client connection negotiation failed: client connection to 127.16.233.124:35937: connect: Connection refused (error 111)
I20260810 22:38:43.193967 9997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:38:43.194183 9997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:38:43.194267 9907 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.233.124" port: 35937 } attempt: 2
I20260810 22:38:43.195030 9910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:43.194721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.124:35937 (local address 127.0.0.1:42848)
0810 22:38:43.194859 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:43.194946 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.124:35937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260810 22:38:43.195842 9910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:43.195589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.124:35937 (local address 127.0.0.1:42862)
0810 22:38:43.195713 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:43.195779 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.124:35937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260810 22:38:43.195966 9907 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.233.124:35937: Network error: Client connection negotiation failed: client connection to 127.16.233.124:35937: connect: Connection refused (error 111)
I20260810 22:38:43.204329 9997 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.124:35937
I20260810 22:38:43.204684 10071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.124:35937 every 8 connection(s)
I20260810 22:38:43.205611 9997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/minicluster-data/master-2/data/info.pb
I20260810 22:38:43.209178 10073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:38:43.214133 10073 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.233.126" port: 37779 } has no permanent_uuid. Determining permanent_uuid...
I20260810 22:38:43.214787 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 9997
I20260810 22:38:43.214962 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786401215402979-17317-0/minicluster-data/master-2/wal/instance
I20260810 22:38:43.215286 17317 external_mini_cluster.cc:1428] Setting key e6c5cb9b8274542b15659fbb72b2aeba
Aug 10 22:38:43 dist-test-slave-77v9 krb5kdc[9819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.124@KRBTEST.COM
Aug 10 22:38:43 dist-test-slave-77v9 krb5kdc[9819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401523, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.124@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:38:43.231659 9911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:43.216516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46112 (local address 127.16.233.126:37779)
0810 22:38:43.216698 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:43.216705 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:43.217649 (+ 944us) server_negotiation.cc:407] Connection header received
0810 22:38:43.219059 (+ 1410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:43.219062 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:43.219116 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:43.219202 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:43.220319 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:43.221007 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:43.222264 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:43.222521 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:43.222615 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:43.226985 (+ 4370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:43.226990 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:43.226992 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:38:43.227033 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:43.229107 (+ 2074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:43.229768 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:43.229772 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:43.229774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:43.229893 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:43.230388 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:43.230391 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:43.230413 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:43.230681 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:43.230788 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:43.231330 (+ 542us) server_negotiation.cc:299] Negotiation successful
0810 22:38:43.231485 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 22:38:43.232695 10079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:43.217821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42878 (local address 127.16.233.124:35937)
0810 22:38:43.218139 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:43.218152 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:43.218198 (+ 46us) server_negotiation.cc:407] Connection header received
0810 22:38:43.218758 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:43.218784 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:43.219073 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:43.219304 (+ 231us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:43.220188 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:43.221387 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:43.222360 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:43.223026 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:43.223253 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:43.226257 (+ 3004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:43.226264 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:43.226277 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:38:43.226304 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:43.229101 (+ 2797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:43.229776 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:43.229783 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:43.229785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:43.230007 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:43.230447 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:43.230453 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:43.230457 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:43.231049 (+ 592us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:43.231247 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:43.231590 (+ 343us) server_negotiation.cc:299] Negotiation successful
0810 22:38:43.231751 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":86,"server-negotiator.queue_time_us":207,"spinlock_wait_cycles":1920,"thread_start_us":94,"threads_started":1}
I20260810 22:38:43.232695 10078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:43.216969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:37779 (local address 127.0.0.1:46112)
0810 22:38:43.217481 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:43.217532 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 22:38:43.218751 (+ 1219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:38:43.219298 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:38:43.219304 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:38:43.219821 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:38:43.220135 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:43.220158 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:43.221140 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:43.221145 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:43.222119 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:43.222128 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:43.222905 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:43.222910 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:43.223242 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:43.224134 (+ 892us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:38:43.224153 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:38:43.226639 (+ 2486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:38:43.229258 (+ 2619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:38:43.229271 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:38:43.229285 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:38:43.229635 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:38:43.230030 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:38:43.230032 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:38:43.230034 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:38:43.230246 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:38:43.230836 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:38:43.230842 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:38:43.231028 (+ 186us) client_negotiation.cc:769] Sending connection context
0810 22:38:43.231233 (+ 205us) client_negotiation.cc:240] Negotiation successful
0810 22:38:43.231632 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":51,"thread_start_us":127,"threads_started":1}
I20260810 22:38:43.234632 10073 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.233.125" port: 33363 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 22:38:43 dist-test-slave-77v9 krb5kdc[9819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401523, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.124@KRBTEST.COM for kudu/127.16.233.125@KRBTEST.COM
Aug 10 22:38:43 dist-test-slave-77v9 krb5kdc[9819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:38:43.244640 10078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:43.235398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.125:33363 (local address 127.0.0.1:41052)
0810 22:38:43.235535 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:43.235553 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 22:38:43.235667 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:38:43.236005 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:38:43.236008 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:38:43.236197 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:38:43.236383 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:43.236402 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:43.237257 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:43.237261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:43.237812 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:43.237820 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:43.238285 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:43.238289 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:43.238394 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:43.239001 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:38:43.239015 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:38:43.240647 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:38:43.242745 (+ 2098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:38:43.242750 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:38:43.242752 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:38:43.243035 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:38:43.243433 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:38:43.243435 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:38:43.243437 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:38:43.243513 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:38:43.244020 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:38:43.244022 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:38:43.244135 (+ 113us) client_negotiation.cc:769] Sending connection context
0810 22:38:43.244249 (+ 114us) client_negotiation.cc:240] Negotiation successful
0810 22:38:43.244422 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260810 22:38:43.244669 9996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:43.235555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41052 (local address 127.16.233.125:33363)
0810 22:38:43.235727 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:43.235733 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:43.235748 (+ 15us) server_negotiation.cc:407] Connection header received
0810 22:38:43.235796 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:43.235798 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:43.235865 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:43.235952 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:43.236528 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:43.237138 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:43.237950 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:43.238163 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:43.238271 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:43.240835 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:43.240839 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:43.240845 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:38:43.240884 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:43.242618 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:43.243181 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:43.243185 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:43.243187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:43.243332 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:43.243637 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:43.243641 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:43.243642 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:43.243900 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:43.244017 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:43.244362 (+ 345us) server_negotiation.cc:299] Negotiation successful
0810 22:38:43.244491 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 22:38:43.245208 9911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:43.235952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46128 (local address 127.16.233.126:37779)
0810 22:38:43.236126 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:43.236129 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:43.236139 (+ 10us) server_negotiation.cc:407] Connection header received
0810 22:38:43.236172 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:43.236174 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:43.236209 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:43.236270 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:43.237195 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:43.237707 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:43.238500 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:43.238836 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:43.238951 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:43.241368 (+ 2417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:43.241372 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:43.241375 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:38:43.241411 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:43.243061 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:43.243605 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:43.243610 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:43.243615 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:43.243699 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:43.244019 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:43.244022 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:43.244024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:43.244259 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:43.244498 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:43.244627 (+ 129us) server_negotiation.cc:299] Negotiation successful
0810 22:38:43.245042 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260810 22:38:43.246222 10073 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.233.124" port: 35937 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 22:38:43 dist-test-slave-77v9 krb5kdc[9819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401523, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.124@KRBTEST.COM for kudu/127.16.233.124@KRBTEST.COM
Aug 10 22:38:43 dist-test-slave-77v9 krb5kdc[9819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.125@KRBTEST.COM
I20260810 22:38:43.257874 10078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:43.246806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.124:35937 (local address 127.0.0.1:42890)
0810 22:38:43.246960 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:43.246974 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:38:43.247053 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:38:43.247302 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:38:43.247304 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:38:43.247487 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:38:43.247615 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:43.247623 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:43.248783 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:43.248787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:43.249856 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:43.249871 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:43.250518 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:43.250522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:43.250613 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:43.251386 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:38:43.251405 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:38:43.253300 (+ 1895us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:38:43.255705 (+ 2405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:38:43.255709 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:38:43.255712 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:38:43.255995 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:38:43.256506 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:38:43.256511 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:38:43.256513 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:38:43.256623 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:38:43.257182 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:38:43.257186 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:38:43.257332 (+ 146us) client_negotiation.cc:769] Sending connection context
0810 22:38:43.257455 (+ 123us) client_negotiation.cc:240] Negotiation successful
0810 22:38:43.257665 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260810 22:38:43.258147 10079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:43.246953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42890 (local address 127.16.233.124:35937)
0810 22:38:43.247075 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:43.247082 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:43.247097 (+ 15us) server_negotiation.cc:407] Connection header received
0810 22:38:43.247146 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:43.247149 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:43.247190 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:43.247262 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:43.247735 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:43.248581 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:43.249981 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:43.250399 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:43.250491 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:43.253498 (+ 3007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:43.253503 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:43.253506 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:38:43.253562 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:43.255553 (+ 1991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:43.256149 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:43.256152 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:43.256153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:43.256327 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:43.256739 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:43.256742 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:43.256744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:43.257072 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:43.257197 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:43.257519 (+ 322us) server_negotiation.cc:299] Negotiation successful
0810 22:38:43.257740 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":95,"server-negotiator.queue_time_us":35}
I20260810 22:38:43.258199 9996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:43.246581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41068 (local address 127.16.233.125:33363)
0810 22:38:43.246738 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:43.246743 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:43.246803 (+ 60us) server_negotiation.cc:407] Connection header received
0810 22:38:43.246967 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:43.246970 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:43.247022 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:43.247097 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:43.247929 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:43.248646 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:43.249972 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:43.250328 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:43.250465 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:43.254195 (+ 3730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:43.254218 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:43.254222 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:38:43.254253 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:43.256125 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:43.256625 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:43.256629 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:43.256631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:43.256721 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:43.257083 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:43.257087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:43.257089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:43.257301 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:43.257600 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:43.257690 (+ 90us) server_negotiation.cc:299] Negotiation successful
0810 22:38:43.257827 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 22:38:43.262432 17317 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 22:38:43.262770 10073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71b0d06fae4b4112aec3d98020bf94f3: Bootstrap starting.
I20260810 22:38:43.263979 9879 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "ecf84eda480f48ec8903315ccf9f81d9"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46128
I20260810 22:38:43.265642 10073 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71b0d06fae4b4112aec3d98020bf94f3: Neither blocks nor log segments found. Creating new log.
I20260810 22:38:43.266601 10073 log.cc:824] T 00000000000000000000000000000000 P 71b0d06fae4b4112aec3d98020bf94f3: Log is configured to *not* fsync() on all Append() calls
I20260810 22:38:43.269306 10073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71b0d06fae4b4112aec3d98020bf94f3: No bootstrap required, opened a new log
I20260810 22:38:43.272718 10073 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71b0d06fae4b4112aec3d98020bf94f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf84eda480f48ec8903315ccf9f81d9" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 37779 } } peers { permanent_uuid: "ae8665eb53d548c8ac38052cdf949b1e" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 33363 } } peers { permanent_uuid: "71b0d06fae4b4112aec3d98020bf94f3" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 35937 } }
I20260810 22:38:43.273142 10073 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71b0d06fae4b4112aec3d98020bf94f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71b0d06fae4b4112aec3d98020bf94f3, State: Initialized, Role: FOLLOWER
I20260810 22:38:43.273531 10073 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71b0d06fae4b4112aec3d98020bf94f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf84eda480f48ec8903315ccf9f81d9" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 37779 } } peers { permanent_uuid: "ae8665eb53d548c8ac38052cdf949b1e" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 33363 } } peers { permanent_uuid: "71b0d06fae4b4112aec3d98020bf94f3" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 35937 } }
I20260810 22:38:43.274206 10082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71b0d06fae4b4112aec3d98020bf94f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf84eda480f48ec8903315ccf9f81d9" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 37779 } } peers { permanent_uuid: "ae8665eb53d548c8ac38052cdf949b1e" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 33363 } } peers { permanent_uuid: "71b0d06fae4b4112aec3d98020bf94f3" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 35937 } } }
I20260810 22:38:43.274394 10082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71b0d06fae4b4112aec3d98020bf94f3 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 22:38:43.274950 10073 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71b0d06fae4b4112aec3d98020bf94f3 [sys.catalog]: configured and running, proceeding with master startup.
W20260810 22:38:43.280547 10094 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 71b0d06fae4b4112aec3d98020bf94f3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 22:38:43.280650 10094 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 22:38:43.279474 9997 master_runner.cc:186] Error getting master registration for 127.16.233.126:37779: OK, instance_id { permanent_uuid: "ecf84eda480f48ec8903315ccf9f81d9" instance_seqno: 1786401522654902 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 22:38:43.283389 9997 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 22:38:43.296142 9907 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.233.124" port: 35937 } attempt: 3
Aug 10 22:38:43 dist-test-slave-77v9 krb5kdc[9819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401522, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.126@KRBTEST.COM for kudu/127.16.233.124@KRBTEST.COM
I20260810 22:38:43.307423 9910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:43.296933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.124:35937 (local address 127.0.0.1:42898)
0810 22:38:43.297125 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:43.297145 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 22:38:43.297264 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:38:43.297536 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:38:43.297538 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:38:43.297812 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:38:43.297916 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:43.297922 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:43.299027 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:43.299030 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:43.299893 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:43.299903 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:43.300548 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:43.300552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:43.300653 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:43.301341 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:38:43.301358 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:38:43.302936 (+ 1578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:38:43.305402 (+ 2466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:38:43.305410 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:38:43.305413 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:38:43.305709 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:38:43.306104 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:38:43.306107 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:38:43.306109 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:38:43.306208 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:38:43.306815 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:38:43.306818 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:38:43.306938 (+ 120us) client_negotiation.cc:769] Sending connection context
0810 22:38:43.307045 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 22:38:43.307215 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260810 22:38:43.307446 10079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:43.297097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42898 (local address 127.16.233.124:35937)
0810 22:38:43.297277 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:43.297282 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:43.297331 (+ 49us) server_negotiation.cc:407] Connection header received
0810 22:38:43.297382 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:43.297385 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:43.297437 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:43.297516 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:43.298059 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:43.298880 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:43.300042 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:43.300406 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:43.300523 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:43.303156 (+ 2633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:43.303160 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:43.303163 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:38:43.303197 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:43.305156 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:43.305831 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:43.305835 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:43.305837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:43.305948 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:43.306318 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:43.306322 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:43.306324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:43.306679 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:43.306907 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:43.307131 (+ 224us) server_negotiation.cc:299] Negotiation successful
0810 22:38:43.307282 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 22:38:43.311332 9907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecf84eda480f48ec8903315ccf9f81d9: Bootstrap starting.
I20260810 22:38:43.314555 9907 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ecf84eda480f48ec8903315ccf9f81d9: Neither blocks nor log segments found. Creating new log.
I20260810 22:38:43.315508 9907 log.cc:824] T 00000000000000000000000000000000 P ecf84eda480f48ec8903315ccf9f81d9: Log is configured to *not* fsync() on all Append() calls
I20260810 22:38:43.317962 9907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecf84eda480f48ec8903315ccf9f81d9: No bootstrap required, opened a new log
I20260810 22:38:43.320636 9907 raft_consensus.cc:348] T 00000000000000000000000000000000 P ecf84eda480f48ec8903315ccf9f81d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf84eda480f48ec8903315ccf9f81d9" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 37779 } } peers { permanent_uuid: "ae8665eb53d548c8ac38052cdf949b1e" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 33363 } } peers { permanent_uuid: "71b0d06fae4b4112aec3d98020bf94f3" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 35937 } }
I20260810 22:38:43.320986 9907 raft_consensus.cc:729] T 00000000000000000000000000000000 P ecf84eda480f48ec8903315ccf9f81d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecf84eda480f48ec8903315ccf9f81d9, State: Initialized, Role: FOLLOWER
I20260810 22:38:43.321338 9907 consensus_queue.cc:260] T 00000000000000000000000000000000 P ecf84eda480f48ec8903315ccf9f81d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf84eda480f48ec8903315ccf9f81d9" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 37779 } } peers { permanent_uuid: "ae8665eb53d548c8ac38052cdf949b1e" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 33363 } } peers { permanent_uuid: "71b0d06fae4b4112aec3d98020bf94f3" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 35937 } }
I20260810 22:38:43.321893 10097 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecf84eda480f48ec8903315ccf9f81d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf84eda480f48ec8903315ccf9f81d9" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 37779 } } peers { permanent_uuid: "ae8665eb53d548c8ac38052cdf949b1e" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 33363 } } peers { permanent_uuid: "71b0d06fae4b4112aec3d98020bf94f3" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 35937 } } }
I20260810 22:38:43.322081 10097 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecf84eda480f48ec8903315ccf9f81d9 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 22:38:43.322788 9907 sys_catalog.cc:564] T 00000000000000000000000000000000 P ecf84eda480f48ec8903315ccf9f81d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:38:43.322929 9879 raft_consensus.cc:482] T 00000000000000000000000000000000 P ecf84eda480f48ec8903315ccf9f81d9 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 22:38:43.323113 9879 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ecf84eda480f48ec8903315ccf9f81d9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:38:43.324782 9879 raft_consensus.cc:504] T 00000000000000000000000000000000 P ecf84eda480f48ec8903315ccf9f81d9 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf84eda480f48ec8903315ccf9f81d9" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 37779 } } peers { permanent_uuid: "ae8665eb53d548c8ac38052cdf949b1e" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 33363 } } peers { permanent_uuid: "71b0d06fae4b4112aec3d98020bf94f3" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 35937 } }
I20260810 22:38:43.327589 10043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ecf84eda480f48ec8903315ccf9f81d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "71b0d06fae4b4112aec3d98020bf94f3"
I20260810 22:38:43.327916 10043 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71b0d06fae4b4112aec3d98020bf94f3 [term 0 FOLLOWER]: Advancing to term 1
W20260810 22:38:43.330101 10108 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ecf84eda480f48ec8903315ccf9f81d9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 22:38:43.330209 10108 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 22:38:43.331434 10043 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 71b0d06fae4b4112aec3d98020bf94f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecf84eda480f48ec8903315ccf9f81d9 in term 1.
I20260810 22:38:43.331774 9879 leader_election.cc:288] T 00000000000000000000000000000000 P ecf84eda480f48ec8903315ccf9f81d9 [CANDIDATE]: Term 1 election: Requested vote from peers ae8665eb53d548c8ac38052cdf949b1e (127.16.233.125:33363), 71b0d06fae4b4112aec3d98020bf94f3 (127.16.233.124:35937)
I20260810 22:38:43.332571 9851 leader_election.cc:302] T 00000000000000000000000000000000 P ecf84eda480f48ec8903315ccf9f81d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71b0d06fae4b4112aec3d98020bf94f3, ecf84eda480f48ec8903315ccf9f81d9; no voters:
I20260810 22:38:43.332993 10097 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ecf84eda480f48ec8903315ccf9f81d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:38:43.333256 10097 raft_consensus.cc:686] T 00000000000000000000000000000000 P ecf84eda480f48ec8903315ccf9f81d9 [term 1 LEADER]: Becoming Leader. State: Replica: ecf84eda480f48ec8903315ccf9f81d9, State: Running, Role: LEADER
I20260810 22:38:43.330143 9833 master_runner.cc:186] Error getting master registration for 127.16.233.125:33363: OK, instance_id { permanent_uuid: "ae8665eb53d548c8ac38052cdf949b1e" instance_seqno: 1786401522930086 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 22:38:43.333658 10097 consensus_queue.cc:237] T 00000000000000000000000000000000 P ecf84eda480f48ec8903315ccf9f81d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf84eda480f48ec8903315ccf9f81d9" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 37779 } } peers { permanent_uuid: "ae8665eb53d548c8ac38052cdf949b1e" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 33363 } } peers { permanent_uuid: "71b0d06fae4b4112aec3d98020bf94f3" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 35937 } }
I20260810 22:38:43.333777 9833 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 22:38:43.336989 9991 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.233.124" port: 35937 } attempt: 4
I20260810 22:38:43.338415 10112 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecf84eda480f48ec8903315ccf9f81d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ecf84eda480f48ec8903315ccf9f81d9. Latest consensus state: current_term: 1 leader_uuid: "ecf84eda480f48ec8903315ccf9f81d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf84eda480f48ec8903315ccf9f81d9" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 37779 } } peers { permanent_uuid: "ae8665eb53d548c8ac38052cdf949b1e" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 33363 } } peers { permanent_uuid: "71b0d06fae4b4112aec3d98020bf94f3" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 35937 } } }
I20260810 22:38:43.338613 10112 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecf84eda480f48ec8903315ccf9f81d9 [sys.catalog]: This master's current role is: LEADER
I20260810 22:38:43.340296 10122 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:38:43.341506 10122 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 10 22:38:43 dist-test-slave-77v9 krb5kdc[9819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401522, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.125@KRBTEST.COM for kudu/127.16.233.124@KRBTEST.COM
I20260810 22:38:43.355444 10079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:43.334538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42910 (local address 127.16.233.124:35937)
0810 22:38:43.334669 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:43.334674 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:43.340131 (+ 5457us) server_negotiation.cc:407] Connection header received
0810 22:38:43.340182 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:43.340185 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:43.340233 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:43.340323 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:43.345346 (+ 5023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:43.346141 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:43.347435 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:43.347791 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:43.347896 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:43.350710 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:43.350715 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:43.350718 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:38:43.350754 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:43.353028 (+ 2274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:43.353645 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:43.353649 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:43.353651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:43.353755 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:43.354160 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:43.354165 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:43.354168 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:43.354453 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:43.354616 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:43.355043 (+ 427us) server_negotiation.cc:299] Negotiation successful
0810 22:38:43.355200 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 22:38:43.359929 9996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:43.335033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41080 (local address 127.16.233.125:33363)
0810 22:38:43.335151 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:43.335156 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:43.339134 (+ 3978us) server_negotiation.cc:407] Connection header received
0810 22:38:43.339361 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:43.339365 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:43.339416 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:43.339471 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:43.343568 (+ 4097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:43.344476 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:43.347941 (+ 3465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:43.348336 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:43.348429 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:43.353585 (+ 5156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:43.353589 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:43.353593 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:38:43.353624 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:43.356102 (+ 2478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:43.358183 (+ 2081us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:43.358189 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:43.358192 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:43.358298 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:43.358821 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:43.358826 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:43.358830 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:43.359086 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:43.359230 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:43.359604 (+ 374us) server_negotiation.cc:299] Negotiation successful
0810 22:38:43.359732 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 22:38:43.363199 9994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:43.337515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.124:35937 (local address 127.0.0.1:42922)
0810 22:38:43.337793 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:43.337810 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 22:38:43.337914 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:38:43.339613 (+ 1699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:38:43.339616 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:38:43.339883 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:38:43.340001 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:43.340009 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:43.345272 (+ 5263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:43.345276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:43.346255 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:43.346268 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:43.348622 (+ 2354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:43.348626 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:43.348719 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:43.349166 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:38:43.349183 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:38:43.351299 (+ 2116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:38:43.360862 (+ 9563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:38:43.360868 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:38:43.360870 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:38:43.361155 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:38:43.361739 (+ 584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:38:43.361744 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:38:43.361746 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:38:43.361864 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:38:43.362605 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:38:43.362611 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:38:43.362725 (+ 114us) client_negotiation.cc:769] Sending connection context
0810 22:38:43.362832 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 22:38:43.363010 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200}
I20260810 22:38:43.363221 10121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:43.338097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42922 (local address 127.16.233.124:35937)
0810 22:38:43.339324 (+ 1227us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:43.339334 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:43.339351 (+ 17us) server_negotiation.cc:407] Connection header received
0810 22:38:43.339412 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:43.339414 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:43.339472 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:43.339580 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:43.342412 (+ 2832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:43.343403 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:43.347157 (+ 3754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:43.347601 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:43.349096 (+ 1495us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:43.352033 (+ 2937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:43.352037 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:43.352040 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:38:43.352060 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:43.360671 (+ 8611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:43.361501 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:43.361506 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:43.361509 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:43.361609 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:43.362206 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:43.362210 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:43.362211 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:43.362497 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:43.362596 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:43.362905 (+ 309us) server_negotiation.cc:299] Negotiation successful
0810 22:38:43.363039 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1118,"thread_start_us":114,"threads_started":1}
I20260810 22:38:43.365008 9911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:43.336852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46140 (local address 127.16.233.126:37779)
0810 22:38:43.337006 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:43.337013 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:43.341850 (+ 4837us) server_negotiation.cc:407] Connection header received
0810 22:38:43.341976 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:43.341981 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:43.342045 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:43.342141 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:43.348947 (+ 6806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:43.349860 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:43.351459 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:43.351825 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:43.351939 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:43.354827 (+ 2888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:43.354840 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:43.354843 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:38:43.354878 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:43.357656 (+ 2778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:43.362211 (+ 4555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:43.362216 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:43.362218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:43.362421 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:43.363666 (+ 1245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:43.363669 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:43.363671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:43.363948 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:43.364096 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:43.364606 (+ 510us) server_negotiation.cc:299] Negotiation successful
0810 22:38:43.364767 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 22:38:43.367271 9991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae8665eb53d548c8ac38052cdf949b1e: Bootstrap starting.
I20260810 22:38:43.367808 10043 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 71b0d06fae4b4112aec3d98020bf94f3 [term 1 FOLLOWER]: Refusing update from remote peer ecf84eda480f48ec8903315ccf9f81d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:38:43.368901 10097 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ecf84eda480f48ec8903315ccf9f81d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71b0d06fae4b4112aec3d98020bf94f3" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 35937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:38:43.373142 9991 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae8665eb53d548c8ac38052cdf949b1e: Neither blocks nor log segments found. Creating new log.
I20260810 22:38:43.375300 9991 log.cc:824] T 00000000000000000000000000000000 P ae8665eb53d548c8ac38052cdf949b1e: Log is configured to *not* fsync() on all Append() calls
I20260810 22:38:43.378052 10082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71b0d06fae4b4112aec3d98020bf94f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ecf84eda480f48ec8903315ccf9f81d9. Latest consensus state: current_term: 1 leader_uuid: "ecf84eda480f48ec8903315ccf9f81d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf84eda480f48ec8903315ccf9f81d9" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 37779 } } peers { permanent_uuid: "ae8665eb53d548c8ac38052cdf949b1e" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 33363 } } peers { permanent_uuid: "71b0d06fae4b4112aec3d98020bf94f3" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 35937 } } }
I20260810 22:38:43.378237 10082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71b0d06fae4b4112aec3d98020bf94f3 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 22:38:43.382423 9991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae8665eb53d548c8ac38052cdf949b1e: No bootstrap required, opened a new log
I20260810 22:38:43.383632 10112 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecf84eda480f48ec8903315ccf9f81d9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ecf84eda480f48ec8903315ccf9f81d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf84eda480f48ec8903315ccf9f81d9" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 37779 } } peers { permanent_uuid: "ae8665eb53d548c8ac38052cdf949b1e" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 33363 } } peers { permanent_uuid: "71b0d06fae4b4112aec3d98020bf94f3" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 35937 } } }
I20260810 22:38:43.383793 10112 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecf84eda480f48ec8903315ccf9f81d9 [sys.catalog]: This master's current role is: LEADER
I20260810 22:38:43.383965 10123 mvcc.cc:205] Tried to move back new op lower bound from 7317100639692435456 to 7317100639578959872. Current Snapshot: MvccSnapshot[applied={T|T < 7317100639692435456}]
I20260810 22:38:43.385627 10082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71b0d06fae4b4112aec3d98020bf94f3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ecf84eda480f48ec8903315ccf9f81d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf84eda480f48ec8903315ccf9f81d9" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 37779 } } peers { permanent_uuid: "ae8665eb53d548c8ac38052cdf949b1e" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 33363 } } peers { permanent_uuid: "71b0d06fae4b4112aec3d98020bf94f3" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 35937 } } }
I20260810 22:38:43.385779 10082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71b0d06fae4b4112aec3d98020bf94f3 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 22:38:43.391494 9991 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae8665eb53d548c8ac38052cdf949b1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf84eda480f48ec8903315ccf9f81d9" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 37779 } } peers { permanent_uuid: "ae8665eb53d548c8ac38052cdf949b1e" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 33363 } } peers { permanent_uuid: "71b0d06fae4b4112aec3d98020bf94f3" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 35937 } }
I20260810 22:38:43.392037 9991 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae8665eb53d548c8ac38052cdf949b1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae8665eb53d548c8ac38052cdf949b1e, State: Initialized, Role: FOLLOWER
I20260810 22:38:43.392728 9991 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae8665eb53d548c8ac38052cdf949b1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf84eda480f48ec8903315ccf9f81d9" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 37779 } } peers { permanent_uuid: "ae8665eb53d548c8ac38052cdf949b1e" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 33363 } } peers { permanent_uuid: "71b0d06fae4b4112aec3d98020bf94f3" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 35937 } }
I20260810 22:38:43.393795 10122 catalog_manager.cc:1347] Generated new cluster ID: 979368a9cf004fb8ad5ec2317a7251e2
I20260810 22:38:43.393586 10124 mvcc.cc:205] Tried to move back new op lower bound from 7317100639692435456 to 7317100639578959872. Current Snapshot: MvccSnapshot[applied={T|T < 7317100639692435456}]
I20260810 22:38:43.393885 10122 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:38:43.396127 9959 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae8665eb53d548c8ac38052cdf949b1e [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:38:43.396917 9960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ecf84eda480f48ec8903315ccf9f81d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ae8665eb53d548c8ac38052cdf949b1e"
I20260810 22:38:43.398092 9959 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ae8665eb53d548c8ac38052cdf949b1e [term 1 FOLLOWER]: Refusing update from remote peer ecf84eda480f48ec8903315ccf9f81d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:38:43.398880 10097 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ecf84eda480f48ec8903315ccf9f81d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae8665eb53d548c8ac38052cdf949b1e" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 33363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:38:43.400975 9960 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ae8665eb53d548c8ac38052cdf949b1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecf84eda480f48ec8903315ccf9f81d9 in term 1.
I20260810 22:38:43.406196 9991 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae8665eb53d548c8ac38052cdf949b1e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:38:43.414026 10122 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:38:43.414804 10134 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae8665eb53d548c8ac38052cdf949b1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ecf84eda480f48ec8903315ccf9f81d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf84eda480f48ec8903315ccf9f81d9" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 37779 } } peers { permanent_uuid: "ae8665eb53d548c8ac38052cdf949b1e" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 33363 } } peers { permanent_uuid: "71b0d06fae4b4112aec3d98020bf94f3" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 35937 } } }
I20260810 22:38:43.414983 10134 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae8665eb53d548c8ac38052cdf949b1e [sys.catalog]: This master's current role is: FOLLOWER
I20260810 22:38:43.415100 10134 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae8665eb53d548c8ac38052cdf949b1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ecf84eda480f48ec8903315ccf9f81d9. Latest consensus state: current_term: 1 leader_uuid: "ecf84eda480f48ec8903315ccf9f81d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf84eda480f48ec8903315ccf9f81d9" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 37779 } } peers { permanent_uuid: "ae8665eb53d548c8ac38052cdf949b1e" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 33363 } } peers { permanent_uuid: "71b0d06fae4b4112aec3d98020bf94f3" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 35937 } } }
I20260810 22:38:43.415184 10134 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae8665eb53d548c8ac38052cdf949b1e [sys.catalog]: This master's current role is: FOLLOWER
I20260810 22:38:43.415258 10134 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae8665eb53d548c8ac38052cdf949b1e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ecf84eda480f48ec8903315ccf9f81d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf84eda480f48ec8903315ccf9f81d9" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 37779 } } peers { permanent_uuid: "ae8665eb53d548c8ac38052cdf949b1e" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 33363 } } peers { permanent_uuid: "71b0d06fae4b4112aec3d98020bf94f3" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 35937 } } }
I20260810 22:38:43.415373 10134 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae8665eb53d548c8ac38052cdf949b1e [sys.catalog]: This master's current role is: FOLLOWER
I20260810 22:38:43.415405 10122 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:38:43.417714 10112 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecf84eda480f48ec8903315ccf9f81d9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ecf84eda480f48ec8903315ccf9f81d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf84eda480f48ec8903315ccf9f81d9" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 37779 } } peers { permanent_uuid: "ae8665eb53d548c8ac38052cdf949b1e" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 33363 } } peers { permanent_uuid: "71b0d06fae4b4112aec3d98020bf94f3" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 35937 } } }
I20260810 22:38:43.417870 10112 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecf84eda480f48ec8903315ccf9f81d9 [sys.catalog]: This master's current role is: LEADER
I20260810 22:38:43.421890 10131 mvcc.cc:205] Tried to move back new op lower bound from 7317100639692435456 to 7317100639578959872. Current Snapshot: MvccSnapshot[applied={T|T < 7317100639692435456}]
W20260810 22:38:43.426116 10147 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ae8665eb53d548c8ac38052cdf949b1e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 22:38:43.426235 10147 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 22:38:43.454866 10122 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ecf84eda480f48ec8903315ccf9f81d9: Generated new TSK 0
I20260810 22:38:43.455828 10122 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 22:38:43.495700 9996 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:38:43.492005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41088 (local address 127.16.233.125:33363)
0810 22:38:43.492145 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:43.492150 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:43.494347 (+ 2197us) server_negotiation.cc:407] Connection header received
0810 22:38:43.494607 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:43.494613 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:43.494685 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:43.494748 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:43.495448 (+ 700us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41088: BlockingRecv error: recv got EOF from 127.0.0.1:41088 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260810 22:38:43.497440 10121 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:38:43.492945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42936 (local address 127.16.233.124:35937)
0810 22:38:43.493064 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:43.493069 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:43.496328 (+ 3259us) server_negotiation.cc:407] Connection header received
0810 22:38:43.496561 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:43.496565 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:43.496624 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:43.496678 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:43.497285 (+ 607us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42936: BlockingRecv error: recv got EOF from 127.0.0.1:42936 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260810 22:38:43.499161 9911 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:38:43.493696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46148 (local address 127.16.233.126:37779)
0810 22:38:43.493813 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:43.493817 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:43.497889 (+ 4072us) server_negotiation.cc:407] Connection header received
0810 22:38:43.498307 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:43.498311 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:43.498379 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:43.498435 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:43.499031 (+ 596us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46148: BlockingRecv error: recv got EOF from 127.0.0.1:46148 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260810 22:38:43.504508 9996 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:38:43.502613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41100 (local address 127.16.233.125:33363)
0810 22:38:43.502742 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:43.502747 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:43.503359 (+ 612us) server_negotiation.cc:407] Connection header received
0810 22:38:43.503615 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:43.503618 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:43.503707 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:43.503762 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:43.504383 (+ 621us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41100: BlockingRecv error: recv got EOF from 127.0.0.1:41100 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260810 22:38:43.510337 9996 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:38:43.508399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41112 (local address 127.16.233.125:33363)
0810 22:38:43.508526 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:43.508531 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:43.509199 (+ 668us) server_negotiation.cc:407] Connection header received
0810 22:38:43.509438 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:43.509441 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:43.509496 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:43.509549 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:43.510190 (+ 641us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41112: BlockingRecv error: recv got EOF from 127.0.0.1:41112 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260810 22:38:43.516688 9996 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:38:43.515035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41118 (local address 127.16.233.125:33363)
0810 22:38:43.515215 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:43.515221 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:43.515584 (+ 363us) server_negotiation.cc:407] Connection header received
0810 22:38:43.515775 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:43.515779 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:43.515838 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:43.515928 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:43.516563 (+ 635us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41118: BlockingRecv error: recv got EOF from 127.0.0.1:41118 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260810 22:38:43.525174 9996 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:38:43.523454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41132 (local address 127.16.233.125:33363)
0810 22:38:43.523649 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:43.523670 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:43.523842 (+ 172us) server_negotiation.cc:407] Connection header received
0810 22:38:43.524147 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:43.524152 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:43.524224 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:43.524360 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:43.524998 (+ 638us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41132: BlockingRecv error: recv got EOF from 127.0.0.1:41132 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
W20260810 22:38:43.537273 9996 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:38:43.535661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41142 (local address 127.16.233.125:33363)
0810 22:38:43.535821 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:43.535826 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:43.536039 (+ 213us) server_negotiation.cc:407] Connection header received
0810 22:38:43.536198 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:43.536202 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:43.536281 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:43.536401 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:43.537128 (+ 727us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41142: BlockingRecv error: recv got EOF from 127.0.0.1:41142 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260810 22:38:43.557492 9996 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:38:43.555632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41148 (local address 127.16.233.125:33363)
0810 22:38:43.555767 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:43.555773 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:43.556091 (+ 318us) server_negotiation.cc:407] Connection header received
0810 22:38:43.556244 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:43.556248 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:43.556318 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:43.556433 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:43.557171 (+ 738us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41148: BlockingRecv error: recv got EOF from 127.0.0.1:41148 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260810 22:38:43.593535 9996 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:38:43.591843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41156 (local address 127.16.233.125:33363)
0810 22:38:43.592052 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:43.592058 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:43.592283 (+ 225us) server_negotiation.cc:407] Connection header received
0810 22:38:43.592460 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:43.592464 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:43.592529 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:43.592648 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:43.593386 (+ 738us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41156: BlockingRecv error: recv got EOF from 127.0.0.1:41156 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
W20260810 22:38:43.661924 9996 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:38:43.660527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41162 (local address 127.16.233.125:33363)
0810 22:38:43.660705 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:43.660710 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:43.660968 (+ 258us) server_negotiation.cc:407] Connection header received
0810 22:38:43.661131 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:43.661135 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:43.661195 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:43.661305 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:43.661793 (+ 488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41162: BlockingRecv error: recv got EOF from 127.0.0.1:41162 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260810 22:38:43.794123 9996 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:38:43.792552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41170 (local address 127.16.233.125:33363)
0810 22:38:43.792780 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:43.792786 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:43.793072 (+ 286us) server_negotiation.cc:407] Connection header received
0810 22:38:43.793203 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:43.793207 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:43.793261 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:43.793378 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:43.793987 (+ 609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41170: BlockingRecv error: recv got EOF from 127.0.0.1:41170 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260810 22:38:44.054452 9996 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:38:44.052781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41174 (local address 127.16.233.125:33363)
0810 22:38:44.053138 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:44.053142 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:44.053276 (+ 134us) server_negotiation.cc:407] Connection header received
0810 22:38:44.053470 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:44.053473 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:44.053531 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:44.053621 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:44.054312 (+ 691us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41174: BlockingRecv error: recv got EOF from 127.0.0.1:41174 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
I20260810 22:38:44.282285 10094 catalog_manager.cc:1259] Loaded cluster ID: 979368a9cf004fb8ad5ec2317a7251e2
I20260810 22:38:44.282405 10094 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 71b0d06fae4b4112aec3d98020bf94f3: loading cluster ID for follower catalog manager: success
I20260810 22:38:44.284260 10094 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 71b0d06fae4b4112aec3d98020bf94f3: acquiring CA information for follower catalog manager: success
I20260810 22:38:44.285698 10094 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 71b0d06fae4b4112aec3d98020bf94f3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 22:38:44.428133 10147 catalog_manager.cc:1259] Loaded cluster ID: 979368a9cf004fb8ad5ec2317a7251e2
I20260810 22:38:44.428251 10147 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ae8665eb53d548c8ac38052cdf949b1e: loading cluster ID for follower catalog manager: success
I20260810 22:38:44.430104 10147 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ae8665eb53d548c8ac38052cdf949b1e: acquiring CA information for follower catalog manager: success
I20260810 22:38:44.431455 10147 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ae8665eb53d548c8ac38052cdf949b1e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 22:38:44.573421 10225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:44.568991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41188 (local address 127.16.233.125:33363)
0810 22:38:44.569294 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:44.569300 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:44.569319 (+ 19us) server_negotiation.cc:407] Connection header received
0810 22:38:44.569540 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:44.569544 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:44.569692 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:44.569824 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:38:44.570162 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:44.570725 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:44.571684 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:44.571936 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:44.572061 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:44.572447 (+ 386us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:38:44.572945 (+ 498us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:38:44.573102 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:44.573164 (+ 62us) server_negotiation.cc:299] Negotiation successful
0810 22:38:44.573233 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":88,"threads_started":1}
I20260810 22:38:44.582034 10232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:44.577570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42948 (local address 127.16.233.124:35937)
0810 22:38:44.577848 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:44.577853 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:44.577963 (+ 110us) server_negotiation.cc:407] Connection header received
0810 22:38:44.578074 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:44.578076 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:44.578205 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:44.578375 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:38:44.578588 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:44.579200 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:44.580110 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:44.580341 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:44.580429 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:44.580717 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:38:44.581459 (+ 742us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:38:44.581573 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:44.581674 (+ 101us) server_negotiation.cc:299] Negotiation successful
0810 22:38:44.581733 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":73,"threads_started":1}
I20260810 22:38:44.590569 10225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:44.585915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41204 (local address 127.16.233.125:33363)
0810 22:38:44.586130 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:44.586136 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:44.586745 (+ 609us) server_negotiation.cc:407] Connection header received
0810 22:38:44.586963 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:44.586968 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:44.587035 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:44.587124 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:38:44.587380 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:44.587936 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:44.589101 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:44.589439 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:44.589543 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:44.590012 (+ 469us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:38:44.590117 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:38:44.590172 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:44.590352 (+ 180us) server_negotiation.cc:299] Negotiation successful
0810 22:38:44.590418 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260810 22:38:44.590584 10232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:44.586181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42954 (local address 127.16.233.124:35937)
0810 22:38:44.586298 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:44.586304 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:44.586412 (+ 108us) server_negotiation.cc:407] Connection header received
0810 22:38:44.586565 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:44.586569 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:44.586627 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:44.586701 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:38:44.587008 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:44.587814 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:44.589249 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:44.589610 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:44.589715 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:44.590067 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:38:44.590183 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:38:44.590257 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:44.590376 (+ 119us) server_negotiation.cc:299] Negotiation successful
0810 22:38:44.590443 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 22:38:44.591418 10240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:44.585944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46152 (local address 127.16.233.126:37779)
0810 22:38:44.586298 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:44.586303 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:44.586588 (+ 285us) server_negotiation.cc:407] Connection header received
0810 22:38:44.586758 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:44.586773 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:44.587011 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:44.587137 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:38:44.587528 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:44.588321 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:44.589499 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:44.589811 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:44.589939 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:44.590291 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:38:44.590805 (+ 514us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:38:44.590912 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:44.591069 (+ 157us) server_negotiation.cc:299] Negotiation successful
0810 22:38:44.591255 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":87,"threads_started":1}
I20260810 22:38:44.595641 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 9833
I20260810 22:38:44.606387 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 9915
I20260810 22:38:44.615693 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 9997
2026-08-10T22:38:44Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3673 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:38:44 dist-test-slave-77v9 krb5kdc[10245](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:38:44 dist-test-slave-77v9 krb5kdc[10245](info): set up 2 sockets
Aug 10 22:38:44 dist-test-slave-77v9 krb5kdc[10245](info): commencing operation
krb5kdc: starting...
W20260810 22:38:45.779490 17317 mini_kdc.cc:121] Time spent starting KDC: real 1.136s user 0.000s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/krb5kdc/test-user.keytab.
Aug 10 22:38:45 dist-test-slave-77v9 krb5kdc[10245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:38:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:38:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.233.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab.
Entry for principal kudu/127.16.233.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab.
I20260810 22:38:45.937930 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:36533
--webserver_interface=127.16.233.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab
--principal=kudu/127.16.233.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:46051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.126:36533,127.16.233.125:44069,127.16.233.124:40229
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:38:46.064496 10258 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 22:38:46.064958 10258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:38:46.065034 10258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:38:46.065117 10258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:38:46.069581 10258 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:38:46.069674 10258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:38:46.069694 10258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:38:46.069717 10258 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:38:46.069734 10258 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:38:46.076002 10258 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:46051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.126:36533,127.16.233.125:44069,127.16.233.124:40229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.126.keytab
--principal=kudu/127.16.233.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.126:36533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.233.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:38:46.077425 10258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:38:46.078619 10258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:38:46.088701 10264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:46.088595 10263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:38:46.088933 10258 server_base.cc:1034] running on GCE node
W20260810 22:38:46.088663 10266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:38:46.089712 10258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:38:46.091037 10258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:38:46.092283 10258 hybrid_clock.cc:648] HybridClock initialized: now 1786401526092237 us; error 59 us; skew 500 ppm
Aug 10 22:38:46 dist-test-slave-77v9 krb5kdc[10245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401526, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:38:46.099864 10258 init.cc:377] Logged in from keytab as kudu/127.16.233.126@KRBTEST.COM (short username kudu)
I20260810 22:38:46.101159 10258 webserver.cc:460] Webserver started at http://127.16.233.126:44397/ using document root <none> and password file <none>
I20260810 22:38:46.101818 10258 fs_manager.cc:359] Metadata directory not provided
I20260810 22:38:46.101925 10258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:38:46.102187 10258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:38:46.104457 10258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/minicluster-data/master-0/data/instance:
uuid: "9a17a45865314ce48c232e508e235b6e"
format_stamp: "Formatted at 2026-08-10 22:38:46 on dist-test-slave-77v9"
server_key: "be9cf88bcd1564b131106a2d8fb0f89d"
server_key_iv: "c42c4390a54be56ea95e28b405024460"
server_key_version: "encryptionkey@0"
I20260810 22:38:46.105235 10258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/minicluster-data/master-0/wal/instance:
uuid: "9a17a45865314ce48c232e508e235b6e"
format_stamp: "Formatted at 2026-08-10 22:38:46 on dist-test-slave-77v9"
server_key: "be9cf88bcd1564b131106a2d8fb0f89d"
server_key_iv: "c42c4390a54be56ea95e28b405024460"
server_key_version: "encryptionkey@0"
I20260810 22:38:46.110009 10258 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 22:38:46.113094 10275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:46.114634 10258 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 22:38:46.114867 10258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/minicluster-data/master-0/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/minicluster-data/master-0/wal
uuid: "9a17a45865314ce48c232e508e235b6e"
format_stamp: "Formatted at 2026-08-10 22:38:46 on dist-test-slave-77v9"
server_key: "be9cf88bcd1564b131106a2d8fb0f89d"
server_key_iv: "c42c4390a54be56ea95e28b405024460"
server_key_version: "encryptionkey@0"
I20260810 22:38:46.115031 10258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:38:46.149802 10258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:38:46.153697 10258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:38:46.153959 10258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:38:46.162328 10258 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.126:36533
I20260810 22:38:46.162664 10335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.126:36533 every 8 connection(s)
I20260810 22:38:46.163509 10258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/minicluster-data/master-0/data/info.pb
I20260810 22:38:46.166834 10336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:38:46.167866 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 10258
I20260810 22:38:46.168071 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/minicluster-data/master-0/wal/instance
I20260810 22:38:46.168401 17317 external_mini_cluster.cc:1428] Setting key 94b6d2a1e73f4e9b1b3a4007a59ad2b7
I20260810 22:38:46.171363 10336 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.233.126" port: 36533 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.16.233.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.125@KRBTEST.COM" created.
Aug 10 22:38:46 dist-test-slave-77v9 krb5kdc[10245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401526, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.126@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:38:46.190526 10340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:46.174320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53486 (local address 127.16.233.126:36533)
0810 22:38:46.174835 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:46.174858 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:46.175194 (+ 336us) server_negotiation.cc:407] Connection header received
0810 22:38:46.176447 (+ 1253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:46.176452 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:46.176672 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:46.176914 (+ 242us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:46.177838 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:46.178736 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:46.179808 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:46.180212 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:46.180414 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:46.183815 (+ 3401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:46.183822 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:46.183836 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:38:46.183862 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:46.186393 (+ 2531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:46.187163 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:46.187169 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:46.187170 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:46.187314 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:46.187812 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:46.187815 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:46.187816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:46.188268 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:46.188401 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:46.189080 (+ 679us) server_negotiation.cc:299] Negotiation successful
0810 22:38:46.189321 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":344,"thread_start_us":166,"threads_started":1}
I20260810 22:38:46.190528 10341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:46.174321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:36533 (local address 127.0.0.1:53486)
0810 22:38:46.175011 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:46.175069 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 22:38:46.176084 (+ 1015us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:38:46.176922 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:38:46.176927 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:38:46.177413 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:38:46.177680 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:46.177693 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:46.178885 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:46.178888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:46.179681 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:46.179691 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:46.180393 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:46.180397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:46.180600 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:46.181357 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:38:46.181386 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:38:46.183604 (+ 2218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:38:46.186582 (+ 2978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:38:46.186592 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:38:46.186603 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:38:46.187011 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:38:46.187427 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:38:46.187433 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:38:46.187437 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:38:46.187646 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:38:46.188412 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:38:46.188423 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:38:46.188634 (+ 211us) client_negotiation.cc:769] Sending connection context
0810 22:38:46.188879 (+ 245us) client_negotiation.cc:240] Negotiation successful
0810 22:38:46.189428 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"mutex_wait_us":216,"spinlock_wait_cycles":896,"thread_start_us":216,"threads_started":1}
I20260810 22:38:46.193862 10336 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.233.125" port: 44069 } has no permanent_uuid. Determining permanent_uuid...
I20260810 22:38:46.194789 10341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:46.194474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.125:44069 (local address 127.0.0.1:47668)
0810 22:38:46.194627 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:46.194711 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.125:44069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260810 22:38:46.195072 10277 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.233.125:44069: connect: Connection refused (error 111)
I20260810 22:38:46.196368 10341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:46.196053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.125:44069 (local address 127.0.0.1:47674)
0810 22:38:46.196234 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:46.196296 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.125:44069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260810 22:38:46.196600 10336 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.233.125:44069: Network error: Client connection negotiation failed: client connection to 127.16.233.125:44069: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.125.keytab.
Entry for principal kudu/127.16.233.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.125.keytab.
I20260810 22:38:46.205914 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.125:44069
--webserver_interface=127.16.233.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.125.keytab
--principal=kudu/127.16.233.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:46051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.126:36533,127.16.233.125:44069,127.16.233.124:40229
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 22:38:46.245932 10336 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.233.125" port: 44069 } attempt: 1
I20260810 22:38:46.247025 10341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:46.246661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.125:44069 (local address 127.0.0.1:47688)
0810 22:38:46.246860 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:46.246929 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.125:44069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260810 22:38:46.248198 10341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:46.247886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.125:44069 (local address 127.0.0.1:47694)
0810 22:38:46.248058 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:46.248128 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.125:44069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260810 22:38:46.248344 10336 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.233.125:44069: Network error: Client connection negotiation failed: client connection to 127.16.233.125:44069: connect: Connection refused (error 111)
I20260810 22:38:46.316571 10336 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.233.125" port: 44069 } attempt: 2
I20260810 22:38:46.317831 10341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:46.317420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.125:44069 (local address 127.0.0.1:47700)
0810 22:38:46.317640 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:46.317720 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.125:44069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260810 22:38:46.318830 10341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:46.318544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.125:44069 (local address 127.0.0.1:47702)
0810 22:38:46.318677 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:46.318746 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.125:44069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260810 22:38:46.318981 10336 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.233.125:44069: Network error: Client connection negotiation failed: client connection to 127.16.233.125:44069: connect: Connection refused (error 111)
W20260810 22:38:46.331238 10344 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 22:38:46.331669 10344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:38:46.331718 10344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:38:46.331787 10344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:38:46.336259 10344 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:38:46.336383 10344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:38:46.336422 10344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:38:46.336462 10344 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:38:46.336496 10344 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:38:46.342463 10344 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:46051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.126:36533,127.16.233.125:44069,127.16.233.124:40229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.125.keytab
--principal=kudu/127.16.233.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.125:44069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.233.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:38:46.343801 10344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:38:46.344997 10344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:38:46.354854 10351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:46.354926 10352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:38:46.354899 10344 server_base.cc:1034] running on GCE node
W20260810 22:38:46.355126 10354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:38:46.355806 10344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:38:46.357285 10344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:38:46.358556 10344 hybrid_clock.cc:648] HybridClock initialized: now 1786401526358539 us; error 42 us; skew 500 ppm
Aug 10 22:38:46 dist-test-slave-77v9 krb5kdc[10245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401526, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:38:46.362262 10344 init.cc:377] Logged in from keytab as kudu/127.16.233.125@KRBTEST.COM (short username kudu)
I20260810 22:38:46.363281 10344 webserver.cc:460] Webserver started at http://127.16.233.125:38979/ using document root <none> and password file <none>
I20260810 22:38:46.363863 10344 fs_manager.cc:359] Metadata directory not provided
I20260810 22:38:46.363932 10344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:38:46.364114 10344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:38:46.366641 10344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/minicluster-data/master-1/data/instance:
uuid: "eff5a2103e23408c8d6eac602958b675"
format_stamp: "Formatted at 2026-08-10 22:38:46 on dist-test-slave-77v9"
server_key: "72e78241a69227b9c459b6341fdbde9b"
server_key_iv: "65f5e3fd2262e0b17a02276b7eb98693"
server_key_version: "encryptionkey@0"
I20260810 22:38:46.367331 10344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/minicluster-data/master-1/wal/instance:
uuid: "eff5a2103e23408c8d6eac602958b675"
format_stamp: "Formatted at 2026-08-10 22:38:46 on dist-test-slave-77v9"
server_key: "72e78241a69227b9c459b6341fdbde9b"
server_key_iv: "65f5e3fd2262e0b17a02276b7eb98693"
server_key_version: "encryptionkey@0"
I20260810 22:38:46.371649 10344 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260810 22:38:46.374475 10362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:46.375957 10344 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 22:38:46.376130 10344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/minicluster-data/master-1/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/minicluster-data/master-1/wal
uuid: "eff5a2103e23408c8d6eac602958b675"
format_stamp: "Formatted at 2026-08-10 22:38:46 on dist-test-slave-77v9"
server_key: "72e78241a69227b9c459b6341fdbde9b"
server_key_iv: "65f5e3fd2262e0b17a02276b7eb98693"
server_key_version: "encryptionkey@0"
I20260810 22:38:46.376310 10344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:38:46.402469 10344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:38:46.406809 10344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:38:46.407030 10344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:38:46.410179 10336 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.233.125" port: 44069 } attempt: 3
I20260810 22:38:46.411321 10341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:46.410844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.125:44069 (local address 127.0.0.1:47714)
0810 22:38:46.411083 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:46.411167 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.125:44069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260810 22:38:46.412343 10341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:46.412028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.125:44069 (local address 127.0.0.1:47722)
0810 22:38:46.412210 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:46.412269 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.125:44069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260810 22:38:46.412518 10336 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.233.125:44069: Network error: Client connection negotiation failed: client connection to 127.16.233.125:44069: connect: Connection refused (error 111)
I20260810 22:38:46.416664 10344 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.125:44069
I20260810 22:38:46.417042 10417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.125:44069 every 8 connection(s)
I20260810 22:38:46.418080 10344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/minicluster-data/master-1/data/info.pb
I20260810 22:38:46.421825 10418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:38:46.425865 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 10344
I20260810 22:38:46.426165 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/minicluster-data/master-1/wal/instance
I20260810 22:38:46.426574 17317 external_mini_cluster.cc:1428] Setting key 58cda86b8cb80d93ee739c1e35f1f4b1
I20260810 22:38:46.426510 10418 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.233.126" port: 36533 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.16.233.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.233.124@KRBTEST.COM" created.
Aug 10 22:38:46 dist-test-slave-77v9 krb5kdc[10245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401526, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.125@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:38:46.447081 10340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:46.428785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53500 (local address 127.16.233.126:36533)
0810 22:38:46.429008 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:46.429012 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:46.430030 (+ 1018us) server_negotiation.cc:407] Connection header received
0810 22:38:46.431128 (+ 1098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:46.431134 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:46.431214 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:46.431326 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:46.433183 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:46.434148 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:46.435558 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:46.436011 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:46.436166 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:46.442065 (+ 5899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:46.442074 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:46.442081 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:38:46.442132 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:46.444294 (+ 2162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:46.445193 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:46.445196 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:46.445197 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:46.445300 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:46.445725 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:46.445728 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:46.445729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:46.445952 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:46.446068 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:46.446726 (+ 658us) server_negotiation.cc:299] Negotiation successful
0810 22:38:46.446888 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260810 22:38:46.448057 10421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:46.429224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:36533 (local address 127.0.0.1:53500)
0810 22:38:46.429794 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:46.429860 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0810 22:38:46.430874 (+ 1014us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:38:46.431581 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:38:46.431589 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:38:46.432271 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:38:46.432974 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:46.432994 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:46.434351 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:46.434354 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:46.435401 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:46.435411 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:46.436230 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:46.436232 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:46.436412 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:46.437237 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:38:46.437256 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:38:46.441778 (+ 4522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:38:46.444508 (+ 2730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:38:46.444524 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:38:46.444543 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:38:46.445032 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:38:46.445415 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:38:46.445419 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:38:46.445421 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:38:46.445597 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:38:46.446072 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:38:46.446082 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:38:46.446420 (+ 338us) client_negotiation.cc:769] Sending connection context
0810 22:38:46.446647 (+ 227us) client_negotiation.cc:240] Negotiation successful
0810 22:38:46.447041 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":179,"threads_started":1}
I20260810 22:38:46.450351 10418 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.233.125" port: 44069 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 22:38:46 dist-test-slave-77v9 krb5kdc[10245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401526, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.125@KRBTEST.COM for kudu/127.16.233.125@KRBTEST.COM
I20260810 22:38:46.462013 10421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:46.450988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.125:44069 (local address 127.0.0.1:47728)
0810 22:38:46.451130 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:46.451147 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 22:38:46.451243 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:38:46.451990 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:38:46.451993 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:38:46.452187 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:38:46.452280 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:46.452285 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:46.453358 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:46.453363 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:46.454064 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:46.454075 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:46.454932 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:46.454935 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:46.455021 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:46.455646 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:38:46.455658 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:38:46.457238 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:38:46.459918 (+ 2680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:38:46.459921 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:38:46.459924 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:38:46.460207 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:38:46.460557 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:38:46.460560 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:38:46.460562 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:38:46.460646 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:38:46.461282 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:38:46.461285 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:38:46.461405 (+ 120us) client_negotiation.cc:769] Sending connection context
0810 22:38:46.461557 (+ 152us) client_negotiation.cc:240] Negotiation successful
0810 22:38:46.461723 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260810 22:38:46.462030 10424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:46.451178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47728 (local address 127.16.233.125:44069)
0810 22:38:46.451569 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:46.451581 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:46.451601 (+ 20us) server_negotiation.cc:407] Connection header received
0810 22:38:46.451638 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:46.451642 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:46.451809 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:46.451969 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:46.452392 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:46.453239 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:46.454226 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:46.454796 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:46.454928 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:46.457478 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:46.457502 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:46.457518 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:38:46.457547 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:46.459754 (+ 2207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:46.460356 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:46.460362 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:46.460364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:46.460451 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:46.460780 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:46.460784 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:46.460785 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:46.461161 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:46.461322 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:46.461653 (+ 331us) server_negotiation.cc:299] Negotiation successful
0810 22:38:46.461787 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":182,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.233.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.124.keytab.
Entry for principal kudu/127.16.233.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.124.keytab.
I20260810 22:38:46.464427 10418 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.233.124" port: 40229 } has no permanent_uuid. Determining permanent_uuid...
I20260810 22:38:46.465559 10421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:46.465100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.124:40229 (local address 127.0.0.1:58534)
0810 22:38:46.465276 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:46.465441 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.124:40229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260810 22:38:46.465785 17317 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfcURSo/build/debug/bin/kudu
/tmp/dist-test-taskfcURSo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.124:40229
--webserver_interface=127.16.233.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.124.keytab
--principal=kudu/127.16.233.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.233.84:46051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.233.126:36533,127.16.233.125:44069,127.16.233.124:40229
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:38:46.465873 10366 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.233.124:40229: connect: Connection refused (error 111)
I20260810 22:38:46.467132 10421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:46.466759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.124:40229 (local address 127.0.0.1:58548)
0810 22:38:46.466932 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:46.467013 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.124:40229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260810 22:38:46.467383 10418 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.233.124:40229: Network error: Client connection negotiation failed: client connection to 127.16.233.124:40229: connect: Connection refused (error 111)
I20260810 22:38:46.516876 10418 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.233.124" port: 40229 } attempt: 1
I20260810 22:38:46.518028 10421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:46.517621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.124:40229 (local address 127.0.0.1:58560)
0810 22:38:46.517837 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:46.517930 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.124:40229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260810 22:38:46.519001 10421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:46.518719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.124:40229 (local address 127.0.0.1:58562)
0810 22:38:46.518857 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:46.518932 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.124:40229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260810 22:38:46.519207 10418 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.233.124:40229: Network error: Client connection negotiation failed: client connection to 127.16.233.124:40229: connect: Connection refused (error 111)
I20260810 22:38:46.555730 10336 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.233.125" port: 44069 } attempt: 4
Aug 10 22:38:46 dist-test-slave-77v9 krb5kdc[10245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401526, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.126@KRBTEST.COM for kudu/127.16.233.125@KRBTEST.COM
I20260810 22:38:46.566967 10424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:46.556654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47734 (local address 127.16.233.125:44069)
0810 22:38:46.556903 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:46.556909 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:46.556929 (+ 20us) server_negotiation.cc:407] Connection header received
0810 22:38:46.557091 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:46.557094 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:46.557166 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:46.557271 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:46.557794 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:46.559086 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:46.560556 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:46.560933 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:46.561056 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:46.563398 (+ 2342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:46.563402 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:46.563405 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:38:46.563442 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:46.565132 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:46.565639 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:46.565644 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:46.565645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:46.565734 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:46.566042 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:46.566047 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:46.566049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:46.566259 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:46.566373 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:46.566661 (+ 288us) server_negotiation.cc:299] Negotiation successful
0810 22:38:46.566799 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 22:38:46.566967 10341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:46.556507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.125:44069 (local address 127.0.0.1:47734)
0810 22:38:46.556787 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:46.556832 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 22:38:46.556959 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:38:46.557322 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:38:46.557325 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:38:46.557530 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:38:46.557650 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:46.557657 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:46.559218 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:46.559224 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:46.560401 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:46.560413 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:46.561067 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:46.561070 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:46.561184 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:46.561786 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:38:46.561806 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:38:46.563224 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:38:46.565285 (+ 2061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:38:46.565288 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:38:46.565290 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:38:46.565527 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:38:46.565845 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:38:46.565849 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:38:46.565851 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:38:46.565947 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:38:46.566365 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:38:46.566368 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:38:46.566488 (+ 120us) client_negotiation.cc:769] Sending connection context
0810 22:38:46.566607 (+ 119us) client_negotiation.cc:240] Negotiation successful
0810 22:38:46.566783 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170}
I20260810 22:38:46.568186 10336 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.233.124" port: 40229 } has no permanent_uuid. Determining permanent_uuid...
I20260810 22:38:46.569082 10341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:46.568757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.124:40229 (local address 127.0.0.1:58568)
0810 22:38:46.568947 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:46.569005 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.124:40229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260810 22:38:46.569818 10341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:46.569591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.124:40229 (local address 127.0.0.1:58578)
0810 22:38:46.569699 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:46.569749 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.124:40229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260810 22:38:46.569943 10336 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.233.124:40229: Network error: Client connection negotiation failed: client connection to 127.16.233.124:40229: connect: Connection refused (error 111)
I20260810 22:38:46.587406 10418 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.233.124" port: 40229 } attempt: 2
I20260810 22:38:46.588534 10421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:46.588174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.124:40229 (local address 127.0.0.1:58580)
0810 22:38:46.588360 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:46.588441 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.124:40229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260810 22:38:46.589407 10421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:46.589164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.124:40229 (local address 127.0.0.1:58596)
0810 22:38:46.589283 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:46.589343 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.124:40229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 22:38:46.589532 10418 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.233.124:40229: Network error: Client connection negotiation failed: client connection to 127.16.233.124:40229: connect: Connection refused (error 111)
W20260810 22:38:46.596431 10426 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 22:38:46.596884 10426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:38:46.596958 10426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:38:46.597059 10426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:38:46.601696 10426 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:38:46.601825 10426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:38:46.601861 10426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:38:46.601902 10426 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:38:46.601943 10426 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:38:46.608136 10426 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.233.84:46051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.233.126:36533,127.16.233.125:44069,127.16.233.124:40229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/krb5kdc/kudu_127.16.233.124.keytab
--principal=kudu/127.16.233.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.233.124:40229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.233.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:38:46.609566 10426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:38:46.610736 10426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:38:46.620790 10431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:46.620851 10434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:38:46.620790 10432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:38:46.621140 10426 server_base.cc:1034] running on GCE node
I20260810 22:38:46.622074 10426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:38:46.623901 10426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:38:46.625130 10426 hybrid_clock.cc:648] HybridClock initialized: now 1786401526625122 us; error 58 us; skew 500 ppm
Aug 10 22:38:46 dist-test-slave-77v9 krb5kdc[10245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401526, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:38:46.628667 10426 init.cc:377] Logged in from keytab as kudu/127.16.233.124@KRBTEST.COM (short username kudu)
I20260810 22:38:46.629137 10336 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.233.124" port: 40229 } attempt: 1
I20260810 22:38:46.630295 10341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:46.629751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.124:40229 (local address 127.0.0.1:58608)
0810 22:38:46.630062 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:46.630139 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.124:40229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189}
I20260810 22:38:46.630304 10426 webserver.cc:460] Webserver started at http://127.16.233.124:39155/ using document root <none> and password file <none>
I20260810 22:38:46.631145 10341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:46.630871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.124:40229 (local address 127.0.0.1:58616)
0810 22:38:46.631004 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:46.631079 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.233.124:40229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260810 22:38:46.631132 10426 fs_manager.cc:359] Metadata directory not provided
I20260810 22:38:46.631244 10426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
W20260810 22:38:46.631280 10336 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.233.124:40229: Network error: Client connection negotiation failed: client connection to 127.16.233.124:40229: connect: Connection refused (error 111)
I20260810 22:38:46.631510 10426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:38:46.634229 10426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/minicluster-data/master-2/data/instance:
uuid: "cadb66c9079346e6bf77d976055f22f3"
format_stamp: "Formatted at 2026-08-10 22:38:46 on dist-test-slave-77v9"
server_key: "680514e1663557d8027e5854e7a4be6a"
server_key_iv: "fbf77fbf56327edefbddc1ed61b7bc69"
server_key_version: "encryptionkey@0"
I20260810 22:38:46.635071 10426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/minicluster-data/master-2/wal/instance:
uuid: "cadb66c9079346e6bf77d976055f22f3"
format_stamp: "Formatted at 2026-08-10 22:38:46 on dist-test-slave-77v9"
server_key: "680514e1663557d8027e5854e7a4be6a"
server_key_iv: "fbf77fbf56327edefbddc1ed61b7bc69"
server_key_version: "encryptionkey@0"
I20260810 22:38:46.640305 10426 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260810 22:38:46.643627 10445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:38:46.645332 10426 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:38:46.645540 10426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/minicluster-data/master-2/data,/tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/minicluster-data/master-2/wal
uuid: "cadb66c9079346e6bf77d976055f22f3"
format_stamp: "Formatted at 2026-08-10 22:38:46 on dist-test-slave-77v9"
server_key: "680514e1663557d8027e5854e7a4be6a"
server_key_iv: "fbf77fbf56327edefbddc1ed61b7bc69"
server_key_version: "encryptionkey@0"
I20260810 22:38:46.645694 10426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:38:46.662135 10426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:38:46.666391 10426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:38:46.666687 10426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:38:46.674700 10426 rpc_server.cc:307] RPC server started. Bound to: 127.16.233.124:40229
I20260810 22:38:46.675102 10501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.233.124:40229 every 8 connection(s)
I20260810 22:38:46.675844 10426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/minicluster-data/master-2/data/info.pb
I20260810 22:38:46.679052 10502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:38:46.680729 10418 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.233.124" port: 40229 } attempt: 3
I20260810 22:38:46.683938 10502 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.233.126" port: 36533 } has no permanent_uuid. Determining permanent_uuid...
I20260810 22:38:46.685673 17317 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfcURSo/build/debug/bin/kudu as pid 10426
I20260810 22:38:46.685843 17317 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfcURSo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786401215402979-17317-0/minicluster-data/master-2/wal/instance
I20260810 22:38:46.686160 17317 external_mini_cluster.cc:1428] Setting key 422f3ecb4c1f7df22854727ecd8e9440
Aug 10 22:38:46 dist-test-slave-77v9 krb5kdc[10245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401526, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.125@KRBTEST.COM for kudu/127.16.233.124@KRBTEST.COM
Aug 10 22:38:46 dist-test-slave-77v9 krb5kdc[10245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401526, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.124@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
Aug 10 22:38:46 dist-test-slave-77v9 krb5kdc[10245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.124@KRBTEST.COM
I20260810 22:38:46.696143 10421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:46.681327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.124:40229 (local address 127.0.0.1:58624)
0810 22:38:46.681509 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:46.681529 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 22:38:46.681656 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:38:46.684415 (+ 2759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:38:46.684418 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:38:46.684655 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:38:46.684788 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:46.684796 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:46.686625 (+ 1829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:46.686629 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:46.687460 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:46.687477 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:46.688434 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:46.688438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:46.688525 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:46.689136 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:38:46.689152 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:38:46.690868 (+ 1716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:38:46.693864 (+ 2996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:38:46.693868 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:38:46.693870 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:38:46.694107 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:38:46.694494 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:38:46.694498 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:38:46.694500 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:38:46.694595 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:38:46.695533 (+ 938us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:38:46.695536 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:38:46.695657 (+ 121us) client_negotiation.cc:769] Sending connection context
0810 22:38:46.695779 (+ 122us) client_negotiation.cc:240] Negotiation successful
0810 22:38:46.695963 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260810 22:38:46.697505 10505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:46.682287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58624 (local address 127.16.233.124:40229)
0810 22:38:46.682885 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:46.682918 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:46.682960 (+ 42us) server_negotiation.cc:407] Connection header received
0810 22:38:46.683669 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:46.683689 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:46.684086 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:46.684484 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:46.684988 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:46.686434 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:46.687598 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:46.688299 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:46.688481 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:46.691051 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:46.691058 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:46.691074 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:38:46.691114 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:46.693631 (+ 2517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:46.694240 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:46.694246 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:46.694254 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:46.694383 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:46.694699 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:46.694703 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:46.694704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:46.695407 (+ 703us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:46.695571 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:46.695967 (+ 396us) server_negotiation.cc:299] Negotiation successful
0810 22:38:46.696376 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":186,"threads_started":1}
I20260810 22:38:46.698436 10336 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.233.124" port: 40229 } attempt: 2
I20260810 22:38:46.698882 10506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:46.685904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.126:36533 (local address 127.0.0.1:53504)
0810 22:38:46.686615 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:46.686676 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 22:38:46.686877 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:38:46.687350 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:38:46.687356 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:38:46.687881 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:38:46.688033 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:46.688041 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:46.689155 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:46.689159 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:46.690075 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:46.690085 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:46.690802 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:46.690807 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:46.690931 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:46.691565 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:38:46.691585 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:38:46.693676 (+ 2091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:38:46.696132 (+ 2456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:38:46.696143 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:38:46.696149 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:38:46.696585 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:38:46.696931 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:38:46.696938 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:38:46.696940 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:38:46.697047 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:38:46.697640 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:38:46.697651 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:38:46.697823 (+ 172us) client_negotiation.cc:769] Sending connection context
0810 22:38:46.698465 (+ 642us) client_negotiation.cc:240] Negotiation successful
0810 22:38:46.698689 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":182,"threads_started":1}
I20260810 22:38:46.699960 10507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:46.686824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58634 (local address 127.16.233.124:40229)
0810 22:38:46.687106 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:46.687114 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:46.687368 (+ 254us) server_negotiation.cc:407] Connection header received
0810 22:38:46.687498 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:46.687503 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:46.687559 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:46.687662 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:46.688588 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:46.689310 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:46.690443 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:46.690791 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:46.690898 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:46.694676 (+ 3778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:46.694681 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:46.694683 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:38:46.694709 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:46.697687 (+ 2978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:46.698300 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:46.698304 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:46.698306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:46.698399 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:46.698854 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:46.698858 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:46.698860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:46.699132 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:46.699300 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:46.699646 (+ 346us) server_negotiation.cc:299] Negotiation successful
0810 22:38:46.699784 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"spinlock_wait_cycles":9088,"thread_start_us":96,"threads_started":1}
I20260810 22:38:46.698436 10340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:46.686055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53504 (local address 127.16.233.126:36533)
0810 22:38:46.686203 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:46.686208 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:46.686763 (+ 555us) server_negotiation.cc:407] Connection header received
0810 22:38:46.687022 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:46.687025 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:46.687083 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:46.687144 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:46.688180 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:46.688989 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:46.690212 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:46.690502 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:46.690602 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:46.693888 (+ 3286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:46.693904 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:46.693911 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:38:46.693953 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:46.695966 (+ 2013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:46.696702 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:46.696706 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:46.696708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:46.696798 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:46.697175 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:46.697179 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:46.697181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:46.697462 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:46.697548 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:46.698088 (+ 540us) server_negotiation.cc:299] Negotiation successful
0810 22:38:46.698227 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 22:38:46.702121 10502 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.233.125" port: 44069 } has no permanent_uuid. Determining permanent_uuid...
I20260810 22:38:46.702934 10418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eff5a2103e23408c8d6eac602958b675: Bootstrap starting.
Aug 10 22:38:46 dist-test-slave-77v9 krb5kdc[10245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401526, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.126@KRBTEST.COM for kudu/127.16.233.124@KRBTEST.COM
I20260810 22:38:46.706220 10418 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eff5a2103e23408c8d6eac602958b675: Neither blocks nor log segments found. Creating new log.
I20260810 22:38:46.707304 10418 log.cc:824] T 00000000000000000000000000000000 P eff5a2103e23408c8d6eac602958b675: Log is configured to *not* fsync() on all Append() calls
I20260810 22:38:46.710104 10418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eff5a2103e23408c8d6eac602958b675: No bootstrap required, opened a new log
I20260810 22:38:46.710346 10341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:46.699859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.124:40229 (local address 127.0.0.1:58644)
0810 22:38:46.700260 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:46.700272 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 22:38:46.700333 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:38:46.700850 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:38:46.700853 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:38:46.701042 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:38:46.701149 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:46.701156 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:46.702413 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:46.702418 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:46.703235 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:46.703245 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:46.703824 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:46.703827 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:46.703889 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:46.704517 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:38:46.704529 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:38:46.706262 (+ 1733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:38:46.708393 (+ 2131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:38:46.708397 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:38:46.708399 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:38:46.708666 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:38:46.709035 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:38:46.709040 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:38:46.709044 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:38:46.709150 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:38:46.709695 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:38:46.709698 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:38:46.709816 (+ 118us) client_negotiation.cc:769] Sending connection context
0810 22:38:46.709935 (+ 119us) client_negotiation.cc:240] Negotiation successful
0810 22:38:46.710118 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323}
Aug 10 22:38:46 dist-test-slave-77v9 krb5kdc[10245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401526, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.124@KRBTEST.COM for kudu/127.16.233.125@KRBTEST.COM
I20260810 22:38:46.711141 10507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:46.700450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58644 (local address 127.16.233.124:40229)
0810 22:38:46.700581 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:46.700585 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:46.700597 (+ 12us) server_negotiation.cc:407] Connection header received
0810 22:38:46.700628 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:46.700630 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:46.700677 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:46.700767 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:46.701518 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:46.702277 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:46.703364 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:46.703706 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:46.703817 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:46.706419 (+ 2602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:46.706423 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:46.706426 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:38:46.706469 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:46.708280 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:46.708782 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:46.708784 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:46.708786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:46.708913 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:46.709295 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:46.709298 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:46.709299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:46.709572 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:46.710689 (+ 1117us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:46.710779 (+ 90us) server_negotiation.cc:299] Negotiation successful
0810 22:38:46.710948 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Aug 10 22:38:46 dist-test-slave-77v9 krb5kdc[10245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.126@KRBTEST.COM
I20260810 22:38:46.715240 10336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a17a45865314ce48c232e508e235b6e: Bootstrap starting.
I20260810 22:38:46.715410 10424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:46.704966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47744 (local address 127.16.233.125:44069)
0810 22:38:46.705142 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:46.705149 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:46.705167 (+ 18us) server_negotiation.cc:407] Connection header received
0810 22:38:46.705215 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:46.705221 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:46.705280 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:46.705375 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:46.706022 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:46.706826 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:46.708137 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:46.708510 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:46.708599 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:46.711157 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:46.711163 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:46.711166 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:38:46.711195 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:46.713252 (+ 2057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:46.713821 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:46.713825 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:46.713826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:46.713928 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:46.714268 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:46.714271 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:46.714273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:46.714550 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:46.714627 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:46.715019 (+ 392us) server_negotiation.cc:299] Negotiation successful
0810 22:38:46.715169 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 22:38:46.716200 10506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:46.702643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.125:44069 (local address 127.0.0.1:47744)
0810 22:38:46.702748 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:46.702763 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:38:46.702837 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:38:46.705522 (+ 2685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:38:46.705525 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:38:46.705726 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:38:46.705853 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:46.705859 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:46.707054 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:46.707057 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:46.707979 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:46.707993 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:46.708774 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:46.708778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:46.708913 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:46.709366 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:38:46.709386 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:38:46.710986 (+ 1600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:38:46.713408 (+ 2422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:38:46.713413 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:38:46.713415 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:38:46.713689 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:38:46.714065 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:38:46.714069 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:38:46.714071 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:38:46.714161 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:38:46.714704 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:38:46.714707 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:38:46.714809 (+ 102us) client_negotiation.cc:769] Sending connection context
0810 22:38:46.715721 (+ 912us) client_negotiation.cc:240] Negotiation successful
0810 22:38:46.716023 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260810 22:38:46.716224 10418 raft_consensus.cc:348] T 00000000000000000000000000000000 P eff5a2103e23408c8d6eac602958b675 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a17a45865314ce48c232e508e235b6e" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 36533 } } peers { permanent_uuid: "eff5a2103e23408c8d6eac602958b675" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 44069 } } peers { permanent_uuid: "cadb66c9079346e6bf77d976055f22f3" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 40229 } }
I20260810 22:38:46.716501 10418 raft_consensus.cc:729] T 00000000000000000000000000000000 P eff5a2103e23408c8d6eac602958b675 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff5a2103e23408c8d6eac602958b675, State: Initialized, Role: FOLLOWER
I20260810 22:38:46.716938 10418 consensus_queue.cc:260] T 00000000000000000000000000000000 P eff5a2103e23408c8d6eac602958b675 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a17a45865314ce48c232e508e235b6e" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 36533 } } peers { permanent_uuid: "eff5a2103e23408c8d6eac602958b675" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 44069 } } peers { permanent_uuid: "cadb66c9079346e6bf77d976055f22f3" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 40229 } }
I20260810 22:38:46.717218 10502 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.233.124" port: 40229 } has no permanent_uuid. Determining permanent_uuid...
I20260810 22:38:46.718693 10418 sys_catalog.cc:564] T 00000000000000000000000000000000 P eff5a2103e23408c8d6eac602958b675 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:38:46.718848 10336 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a17a45865314ce48c232e508e235b6e: Neither blocks nor log segments found. Creating new log.
I20260810 22:38:46.720147 10336 log.cc:824] T 00000000000000000000000000000000 P 9a17a45865314ce48c232e508e235b6e: Log is configured to *not* fsync() on all Append() calls
I20260810 22:38:46.720646 10340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:46.701840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53508 (local address 127.16.233.126:36533)
0810 22:38:46.702519 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:46.702522 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:46.705238 (+ 2716us) server_negotiation.cc:407] Connection header received
0810 22:38:46.705456 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:46.705459 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:46.705506 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:46.705556 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:46.707021 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:46.708107 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:46.709596 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:46.709943 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:46.710029 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:46.715135 (+ 5106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:46.715140 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:46.715144 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:38:46.715177 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:46.718026 (+ 2849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:46.718737 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:46.718741 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:46.718742 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:46.718839 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:46.719225 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:46.719229 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:46.719231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:46.719522 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:46.719752 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:46.720268 (+ 516us) server_negotiation.cc:299] Negotiation successful
0810 22:38:46.720447 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605}
I20260810 22:38:46.723086 10336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a17a45865314ce48c232e508e235b6e: No bootstrap required, opened a new log
Aug 10 22:38:46 dist-test-slave-77v9 krb5kdc[10245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401526, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.233.124@KRBTEST.COM for kudu/127.16.233.124@KRBTEST.COM
I20260810 22:38:46.722782 10344 master_runner.cc:186] Error getting master registration for 127.16.233.126:36533: OK, instance_id { permanent_uuid: "9a17a45865314ce48c232e508e235b6e" instance_seqno: 1786401526159932 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 22:38:46.727114 10344 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 22:38:46.727543 10511 sys_catalog.cc:455] T 00000000000000000000000000000000 P eff5a2103e23408c8d6eac602958b675 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a17a45865314ce48c232e508e235b6e" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 36533 } } peers { permanent_uuid: "eff5a2103e23408c8d6eac602958b675" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 44069 } } peers { permanent_uuid: "cadb66c9079346e6bf77d976055f22f3" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 40229 } } }
I20260810 22:38:46.727697 10511 sys_catalog.cc:458] T 00000000000000000000000000000000 P eff5a2103e23408c8d6eac602958b675 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 22:38:46.727816 10336 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a17a45865314ce48c232e508e235b6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a17a45865314ce48c232e508e235b6e" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 36533 } } peers { permanent_uuid: "eff5a2103e23408c8d6eac602958b675" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 44069 } } peers { permanent_uuid: "cadb66c9079346e6bf77d976055f22f3" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 40229 } }
I20260810 22:38:46.728068 10336 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a17a45865314ce48c232e508e235b6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a17a45865314ce48c232e508e235b6e, State: Initialized, Role: FOLLOWER
I20260810 22:38:46.728431 10336 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a17a45865314ce48c232e508e235b6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a17a45865314ce48c232e508e235b6e" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 36533 } } peers { permanent_uuid: "eff5a2103e23408c8d6eac602958b675" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 44069 } } peers { permanent_uuid: "cadb66c9079346e6bf77d976055f22f3" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 40229 } }
I20260810 22:38:46.730002 10336 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a17a45865314ce48c232e508e235b6e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:38:46.731180 10506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:46.717872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.233.124:40229 (local address 127.0.0.1:58650)
0810 22:38:46.718465 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0810 22:38:46.718481 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 22:38:46.718568 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:38:46.721153 (+ 2585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:38:46.721156 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:38:46.721368 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:38:46.721637 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:46.721647 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:46.722973 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:46.722978 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:46.723761 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:38:46.723771 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:38:46.724385 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:38:46.724389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:38:46.724463 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:46.725140 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:38:46.725164 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:38:46.726723 (+ 1559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:38:46.729488 (+ 2765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:38:46.729498 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:38:46.729501 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:38:46.729787 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:38:46.730109 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:38:46.730112 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:38:46.730114 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:38:46.730190 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:38:46.730616 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:38:46.730622 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:38:46.730720 (+ 98us) client_negotiation.cc:769] Sending connection context
0810 22:38:46.730811 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 22:38:46.730977 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":151,"threads_started":1}
W20260810 22:38:46.731232 10527 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eff5a2103e23408c8d6eac602958b675: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 22:38:46.731313 10527 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 22:38:46.731912 10507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:46.718323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58650 (local address 127.16.233.124:40229)
0810 22:38:46.720766 (+ 2443us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:46.720772 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:46.720790 (+ 18us) server_negotiation.cc:407] Connection header received
0810 22:38:46.720864 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:46.720882 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:46.720948 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:46.721035 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:46.721773 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:46.722827 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:46.723914 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:46.724219 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:46.724390 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:46.726919 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:46.726923 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:46.726925 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:38:46.726947 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:46.729338 (+ 2391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:46.729922 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:46.729925 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:46.729927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:46.730009 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:46.730279 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:46.730282 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:46.730284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:46.730499 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:46.731537 (+ 1038us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:46.731618 (+ 81us) server_negotiation.cc:299] Negotiation successful
0810 22:38:46.731741 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2311,"server-negotiator.queue_time_us":2366}
I20260810 22:38:46.732599 10528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a17a45865314ce48c232e508e235b6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a17a45865314ce48c232e508e235b6e" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 36533 } } peers { permanent_uuid: "eff5a2103e23408c8d6eac602958b675" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 44069 } } peers { permanent_uuid: "cadb66c9079346e6bf77d976055f22f3" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 40229 } } }
I20260810 22:38:46.732923 10528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a17a45865314ce48c232e508e235b6e [sys.catalog]: This master's current role is: FOLLOWER
I20260810 22:38:46.735585 10502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cadb66c9079346e6bf77d976055f22f3: Bootstrap starting.
Aug 10 22:38:46 dist-test-slave-77v9 krb5kdc[10245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.233.125@KRBTEST.COM
I20260810 22:38:46.739207 10502 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cadb66c9079346e6bf77d976055f22f3: Neither blocks nor log segments found. Creating new log.
I20260810 22:38:46.740386 10502 log.cc:824] T 00000000000000000000000000000000 P cadb66c9079346e6bf77d976055f22f3: Log is configured to *not* fsync() on all Append() calls
I20260810 22:38:46.738250 10258 master_runner.cc:186] Error getting master registration for 127.16.233.124:40229: OK, instance_id { permanent_uuid: "cadb66c9079346e6bf77d976055f22f3" instance_seqno: 1786401526672439 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 22:38:46.741551 10258 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 22:38:46.742339 10424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:46.727275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47760 (local address 127.16.233.125:44069)
0810 22:38:46.727464 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:46.727470 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:46.727487 (+ 17us) server_negotiation.cc:407] Connection header received
0810 22:38:46.727537 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:46.727541 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:46.727594 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:46.727714 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:46.729553 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:46.730457 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:46.732471 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:46.732986 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:46.733086 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:46.737005 (+ 3919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:46.737009 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:46.737012 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:38:46.737047 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:46.739232 (+ 2185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:46.740015 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:46.740019 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:46.740021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:46.740160 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:46.740705 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:46.740709 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:46.740711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:46.741030 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:46.741241 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:46.742066 (+ 825us) server_negotiation.cc:299] Negotiation successful
0810 22:38:46.742205 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260810 22:38:46.745239 10542 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9a17a45865314ce48c232e508e235b6e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 22:38:46.745345 10542 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 22:38:46.746136 17317 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 22:38:46.747644 10502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cadb66c9079346e6bf77d976055f22f3: No bootstrap required, opened a new log
I20260810 22:38:46.747557 10308 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "9a17a45865314ce48c232e508e235b6e"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53508
I20260810 22:38:46.747870 10308 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a17a45865314ce48c232e508e235b6e [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 22:38:46.748045 10308 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a17a45865314ce48c232e508e235b6e [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:38:46.749677 10308 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a17a45865314ce48c232e508e235b6e [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a17a45865314ce48c232e508e235b6e" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 36533 } } peers { permanent_uuid: "eff5a2103e23408c8d6eac602958b675" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 44069 } } peers { permanent_uuid: "cadb66c9079346e6bf77d976055f22f3" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 40229 } }
I20260810 22:38:46.751536 10390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9a17a45865314ce48c232e508e235b6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "eff5a2103e23408c8d6eac602958b675"
I20260810 22:38:46.751938 10390 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eff5a2103e23408c8d6eac602958b675 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:38:46.752256 10502 raft_consensus.cc:348] T 00000000000000000000000000000000 P cadb66c9079346e6bf77d976055f22f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a17a45865314ce48c232e508e235b6e" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 36533 } } peers { permanent_uuid: "eff5a2103e23408c8d6eac602958b675" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 44069 } } peers { permanent_uuid: "cadb66c9079346e6bf77d976055f22f3" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 40229 } }
I20260810 22:38:46.752555 10502 raft_consensus.cc:729] T 00000000000000000000000000000000 P cadb66c9079346e6bf77d976055f22f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cadb66c9079346e6bf77d976055f22f3, State: Initialized, Role: FOLLOWER
I20260810 22:38:46.752935 10502 consensus_queue.cc:260] T 00000000000000000000000000000000 P cadb66c9079346e6bf77d976055f22f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a17a45865314ce48c232e508e235b6e" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 36533 } } peers { permanent_uuid: "eff5a2103e23408c8d6eac602958b675" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 44069 } } peers { permanent_uuid: "cadb66c9079346e6bf77d976055f22f3" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 40229 } }
I20260810 22:38:46.753734 10390 raft_consensus.cc:2454] T 00000000000000000000000000000000 P eff5a2103e23408c8d6eac602958b675 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a17a45865314ce48c232e508e235b6e in term 1.
I20260810 22:38:46.754344 10277 leader_election.cc:302] T 00000000000000000000000000000000 P 9a17a45865314ce48c232e508e235b6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a17a45865314ce48c232e508e235b6e, eff5a2103e23408c8d6eac602958b675; no voters:
I20260810 22:38:46.754506 10502 sys_catalog.cc:564] T 00000000000000000000000000000000 P cadb66c9079346e6bf77d976055f22f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:38:46.754761 10528 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a17a45865314ce48c232e508e235b6e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:38:46.755035 10528 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a17a45865314ce48c232e508e235b6e [term 1 LEADER]: Becoming Leader. State: Replica: 9a17a45865314ce48c232e508e235b6e, State: Running, Role: LEADER
I20260810 22:38:46.755446 10528 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a17a45865314ce48c232e508e235b6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a17a45865314ce48c232e508e235b6e" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 36533 } } peers { permanent_uuid: "eff5a2103e23408c8d6eac602958b675" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 44069 } } peers { permanent_uuid: "cadb66c9079346e6bf77d976055f22f3" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 40229 } }
I20260810 22:38:46.757716 10528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a17a45865314ce48c232e508e235b6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a17a45865314ce48c232e508e235b6e. Latest consensus state: current_term: 1 leader_uuid: "9a17a45865314ce48c232e508e235b6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a17a45865314ce48c232e508e235b6e" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 36533 } } peers { permanent_uuid: "eff5a2103e23408c8d6eac602958b675" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 44069 } } peers { permanent_uuid: "cadb66c9079346e6bf77d976055f22f3" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 40229 } } }
I20260810 22:38:46.757889 10528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a17a45865314ce48c232e508e235b6e [sys.catalog]: This master's current role is: LEADER
I20260810 22:38:46.757280 10546 sys_catalog.cc:455] T 00000000000000000000000000000000 P cadb66c9079346e6bf77d976055f22f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a17a45865314ce48c232e508e235b6e" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 36533 } } peers { permanent_uuid: "eff5a2103e23408c8d6eac602958b675" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 44069 } } peers { permanent_uuid: "cadb66c9079346e6bf77d976055f22f3" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 40229 } } }
I20260810 22:38:46.758670 10546 sys_catalog.cc:458] T 00000000000000000000000000000000 P cadb66c9079346e6bf77d976055f22f3 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 22:38:46.759310 10474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9a17a45865314ce48c232e508e235b6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "cadb66c9079346e6bf77d976055f22f3"
I20260810 22:38:46.759595 10474 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cadb66c9079346e6bf77d976055f22f3 [term 0 FOLLOWER]: Advancing to term 1
W20260810 22:38:46.761008 10559 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cadb66c9079346e6bf77d976055f22f3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 22:38:46.761094 10559 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 22:38:46.761476 10474 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cadb66c9079346e6bf77d976055f22f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a17a45865314ce48c232e508e235b6e in term 1.
I20260810 22:38:46.763469 10308 leader_election.cc:288] T 00000000000000000000000000000000 P 9a17a45865314ce48c232e508e235b6e [CANDIDATE]: Term 1 election: Requested vote from peers eff5a2103e23408c8d6eac602958b675 (127.16.233.125:44069), cadb66c9079346e6bf77d976055f22f3 (127.16.233.124:40229)
I20260810 22:38:46.766338 10563 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:38:46.770682 10563 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:38:46.778117 10474 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cadb66c9079346e6bf77d976055f22f3 [term 1 FOLLOWER]: Refusing update from remote peer 9a17a45865314ce48c232e508e235b6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:38:46.778972 10547 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9a17a45865314ce48c232e508e235b6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cadb66c9079346e6bf77d976055f22f3" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 40229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:38:46.783551 10546 sys_catalog.cc:455] T 00000000000000000000000000000000 P cadb66c9079346e6bf77d976055f22f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a17a45865314ce48c232e508e235b6e. Latest consensus state: current_term: 1 leader_uuid: "9a17a45865314ce48c232e508e235b6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a17a45865314ce48c232e508e235b6e" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 36533 } } peers { permanent_uuid: "eff5a2103e23408c8d6eac602958b675" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 44069 } } peers { permanent_uuid: "cadb66c9079346e6bf77d976055f22f3" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 40229 } } }
I20260810 22:38:46.783722 10546 sys_catalog.cc:458] T 00000000000000000000000000000000 P cadb66c9079346e6bf77d976055f22f3 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 22:38:46.785045 10390 raft_consensus.cc:1261] T 00000000000000000000000000000000 P eff5a2103e23408c8d6eac602958b675 [term 1 FOLLOWER]: Refusing update from remote peer 9a17a45865314ce48c232e508e235b6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:38:46.785775 10547 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9a17a45865314ce48c232e508e235b6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff5a2103e23408c8d6eac602958b675" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 44069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:38:46.790186 10511 sys_catalog.cc:455] T 00000000000000000000000000000000 P eff5a2103e23408c8d6eac602958b675 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a17a45865314ce48c232e508e235b6e. Latest consensus state: current_term: 1 leader_uuid: "9a17a45865314ce48c232e508e235b6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a17a45865314ce48c232e508e235b6e" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 36533 } } peers { permanent_uuid: "eff5a2103e23408c8d6eac602958b675" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 44069 } } peers { permanent_uuid: "cadb66c9079346e6bf77d976055f22f3" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 40229 } } }
I20260810 22:38:46.790370 10511 sys_catalog.cc:458] T 00000000000000000000000000000000 P eff5a2103e23408c8d6eac602958b675 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 22:38:46.800486 10511 sys_catalog.cc:455] T 00000000000000000000000000000000 P eff5a2103e23408c8d6eac602958b675 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9a17a45865314ce48c232e508e235b6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a17a45865314ce48c232e508e235b6e" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 36533 } } peers { permanent_uuid: "eff5a2103e23408c8d6eac602958b675" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 44069 } } peers { permanent_uuid: "cadb66c9079346e6bf77d976055f22f3" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 40229 } } }
I20260810 22:38:46.800666 10511 sys_catalog.cc:458] T 00000000000000000000000000000000 P eff5a2103e23408c8d6eac602958b675 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 22:38:46.801193 10547 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a17a45865314ce48c232e508e235b6e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9a17a45865314ce48c232e508e235b6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a17a45865314ce48c232e508e235b6e" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 36533 } } peers { permanent_uuid: "eff5a2103e23408c8d6eac602958b675" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 44069 } } peers { permanent_uuid: "cadb66c9079346e6bf77d976055f22f3" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 40229 } } }
I20260810 22:38:46.801360 10547 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a17a45865314ce48c232e508e235b6e [sys.catalog]: This master's current role is: LEADER
I20260810 22:38:46.801510 10573 mvcc.cc:205] Tried to move back new op lower bound from 7317100653671260160 to 7317100653594869760. Current Snapshot: MvccSnapshot[applied={T|T < 7317100653671260160}]
I20260810 22:38:46.802698 10528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a17a45865314ce48c232e508e235b6e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9a17a45865314ce48c232e508e235b6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a17a45865314ce48c232e508e235b6e" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 36533 } } peers { permanent_uuid: "eff5a2103e23408c8d6eac602958b675" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 44069 } } peers { permanent_uuid: "cadb66c9079346e6bf77d976055f22f3" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 40229 } } }
I20260810 22:38:46.802951 10528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a17a45865314ce48c232e508e235b6e [sys.catalog]: This master's current role is: LEADER
I20260810 22:38:46.804342 10546 sys_catalog.cc:455] T 00000000000000000000000000000000 P cadb66c9079346e6bf77d976055f22f3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9a17a45865314ce48c232e508e235b6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a17a45865314ce48c232e508e235b6e" member_type: VOTER last_known_addr { host: "127.16.233.126" port: 36533 } } peers { permanent_uuid: "eff5a2103e23408c8d6eac602958b675" member_type: VOTER last_known_addr { host: "127.16.233.125" port: 44069 } } peers { permanent_uuid: "cadb66c9079346e6bf77d976055f22f3" member_type: VOTER last_known_addr { host: "127.16.233.124" port: 40229 } } }
I20260810 22:38:46.804479 10546 sys_catalog.cc:458] T 00000000000000000000000000000000 P cadb66c9079346e6bf77d976055f22f3 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 22:38:46.805403 10577 mvcc.cc:205] Tried to move back new op lower bound from 7317100653671260160 to 7317100653594869760. Current Snapshot: MvccSnapshot[applied={T|T < 7317100653671260160}]
I20260810 22:38:46.805640 10574 mvcc.cc:205] Tried to move back new op lower bound from 7317100653671260160 to 7317100653594869760. Current Snapshot: MvccSnapshot[applied={T|T < 7317100653671260160}]
I20260810 22:38:46.811439 10563 catalog_manager.cc:1347] Generated new cluster ID: 4b445ebfd2ca428ebb2131fe655ee0b2
I20260810 22:38:46.811527 10563 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:38:46.813755 10424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:46.769294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47772 (local address 127.16.233.125:44069)
0810 22:38:46.769427 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:46.769432 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:46.777424 (+ 7992us) server_negotiation.cc:407] Connection header received
0810 22:38:46.779522 (+ 2098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:46.779526 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:46.779590 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:46.779649 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:46.782533 (+ 2884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:46.783345 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:46.795229 (+ 11884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:46.795604 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:46.795692 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:46.805554 (+ 9862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:46.805558 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:46.805561 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:38:46.805601 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:46.807398 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:46.810791 (+ 3393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:46.810795 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:46.810797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:46.810899 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:46.811418 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:46.811423 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:46.811425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:46.811670 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:46.811816 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:46.813417 (+ 1601us) server_negotiation.cc:299] Negotiation successful
0810 22:38:46.813571 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 22:38:46.815943 10340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:46.770233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53518 (local address 127.16.233.126:36533)
0810 22:38:46.770360 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:46.770365 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:46.777715 (+ 7350us) server_negotiation.cc:407] Connection header received
0810 22:38:46.778103 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:46.778108 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:46.778179 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:46.778246 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:46.783535 (+ 5289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:46.784324 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:46.796580 (+ 12256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:46.797042 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:46.797140 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:46.802984 (+ 5844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:46.802990 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:46.802993 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:38:46.803030 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:46.804916 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:46.810529 (+ 5613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:46.810534 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:46.810536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:46.810654 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:46.812155 (+ 1501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:46.812159 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:46.812161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:46.812419 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:46.812532 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:46.815567 (+ 3035us) server_negotiation.cc:299] Negotiation successful
0810 22:38:46.815736 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 22:38:46.818833 10507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:46.770991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58666 (local address 127.16.233.124:40229)
0810 22:38:46.771117 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:46.771126 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:46.780705 (+ 9579us) server_negotiation.cc:407] Connection header received
0810 22:38:46.780920 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:46.780926 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:46.780978 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:46.781036 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:46.793001 (+ 11965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:46.793848 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:46.797946 (+ 4098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:46.798336 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:46.798424 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:46.808022 (+ 9598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:38:46.808026 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:38:46.808029 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:38:46.808066 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:38:46.809871 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:46.812910 (+ 3039us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:46.812914 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:46.812916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:46.813020 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:38:46.816219 (+ 3199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:38:46.816223 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:38:46.816225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:38:46.816535 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:38:46.816661 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:46.818521 (+ 1860us) server_negotiation.cc:299] Negotiation successful
0810 22:38:46.818664 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 22:38:46.864089 10563 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:38:46.865738 10563 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:38:46.883562 10563 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a17a45865314ce48c232e508e235b6e: Generated new TSK 0
I20260810 22:38:46.884548 10563 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 22:38:46.931183 10340 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:38:46.927047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53522 (local address 127.16.233.126:36533)
0810 22:38:46.927202 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:46.927206 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:46.929546 (+ 2340us) server_negotiation.cc:407] Connection header received
0810 22:38:46.930081 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:46.930086 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:46.930158 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:46.930221 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:46.931016 (+ 795us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53522: BlockingRecv error: recv got EOF from 127.0.0.1:53522 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260810 22:38:46.932583 10507 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:38:46.927998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58668 (local address 127.16.233.124:40229)
0810 22:38:46.928137 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:46.928142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:46.929960 (+ 1818us) server_negotiation.cc:407] Connection header received
0810 22:38:46.931636 (+ 1676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:46.931639 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:46.931701 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:46.931766 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:46.932431 (+ 665us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58668: BlockingRecv error: recv got EOF from 127.0.0.1:58668 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260810 22:38:46.934405 10424 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:38:46.928919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47774 (local address 127.16.233.125:44069)
0810 22:38:46.929062 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:46.929067 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:46.933442 (+ 4375us) server_negotiation.cc:407] Connection header received
0810 22:38:46.933561 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:46.933565 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:46.933634 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:46.933737 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:46.934248 (+ 511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47774: BlockingRecv error: recv got EOF from 127.0.0.1:47774 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260810 22:38:46.939507 10424 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:38:46.937596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47776 (local address 127.16.233.125:44069)
0810 22:38:46.937743 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:46.937747 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:46.938183 (+ 436us) server_negotiation.cc:407] Connection header received
0810 22:38:46.938463 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:46.938468 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:46.938575 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:46.938649 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:46.939345 (+ 696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47776: BlockingRecv error: recv got EOF from 127.0.0.1:47776 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260810 22:38:46.946202 10424 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:38:46.944089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47784 (local address 127.16.233.125:44069)
0810 22:38:46.944249 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:46.944254 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:46.945022 (+ 768us) server_negotiation.cc:407] Connection header received
0810 22:38:46.945273 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:46.945277 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:46.945341 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:46.945411 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:46.946076 (+ 665us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47784: BlockingRecv error: recv got EOF from 127.0.0.1:47784 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260810 22:38:46.953491 10424 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:38:46.951082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47798 (local address 127.16.233.125:44069)
0810 22:38:46.951230 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:46.951237 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:46.952072 (+ 835us) server_negotiation.cc:407] Connection header received
0810 22:38:46.952485 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:46.952490 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:46.952580 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:46.952663 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:46.953358 (+ 695us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47798: BlockingRecv error: recv got EOF from 127.0.0.1:47798 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260810 22:38:46.962730 10424 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:38:46.960527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47808 (local address 127.16.233.125:44069)
0810 22:38:46.960715 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:46.960720 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:46.961456 (+ 736us) server_negotiation.cc:407] Connection header received
0810 22:38:46.961739 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:46.961743 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:46.961822 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:46.961911 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:46.962589 (+ 678us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47808: BlockingRecv error: recv got EOF from 127.0.0.1:47808 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
W20260810 22:38:46.976054 10424 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:38:46.973997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47816 (local address 127.16.233.125:44069)
0810 22:38:46.974133 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:46.974138 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:46.974822 (+ 684us) server_negotiation.cc:407] Connection header received
0810 22:38:46.975082 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:46.975086 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:46.975151 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:46.975211 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:46.975878 (+ 667us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47816: BlockingRecv error: recv got EOF from 127.0.0.1:47816 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260810 22:38:46.996994 10424 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:38:46.995444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47828 (local address 127.16.233.125:44069)
0810 22:38:46.995614 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:46.995621 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:46.995886 (+ 265us) server_negotiation.cc:407] Connection header received
0810 22:38:46.996090 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:46.996093 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:46.996148 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:46.996254 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:46.996798 (+ 544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47828: BlockingRecv error: recv got EOF from 127.0.0.1:47828 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260810 22:38:47.033373 10424 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:38:47.031760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47832 (local address 127.16.233.125:44069)
0810 22:38:47.031982 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:47.031987 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:47.032232 (+ 245us) server_negotiation.cc:407] Connection header received
0810 22:38:47.032470 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:47.032474 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:47.032528 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:47.032628 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:47.033203 (+ 575us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47832: BlockingRecv error: recv got EOF from 127.0.0.1:47832 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
W20260810 22:38:47.101693 10424 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:38:47.100206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47842 (local address 127.16.233.125:44069)
0810 22:38:47.100427 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:47.100432 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:47.100622 (+ 190us) server_negotiation.cc:407] Connection header received
0810 22:38:47.100715 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:47.100718 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:47.100773 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:47.100991 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:47.101515 (+ 524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47842: BlockingRecv error: recv got EOF from 127.0.0.1:47842 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
W20260810 22:38:47.234287 10424 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:38:47.232651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47856 (local address 127.16.233.125:44069)
0810 22:38:47.232919 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:47.232930 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:47.233102 (+ 172us) server_negotiation.cc:407] Connection header received
0810 22:38:47.233307 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:47.233314 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:47.233377 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:47.233483 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:47.234088 (+ 605us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47856: BlockingRecv error: recv got EOF from 127.0.0.1:47856 (error 108)
Metrics: {"server-negotiator.queue_time_us":144}
W20260810 22:38:47.494379 10424 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:38:47.493090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47862 (local address 127.16.233.125:44069)
0810 22:38:47.493329 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:47.493334 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:47.493353 (+ 19us) server_negotiation.cc:407] Connection header received
0810 22:38:47.493493 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:47.493496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:47.493559 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:47.493677 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:38:47.494216 (+ 539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47862: BlockingRecv error: recv got EOF from 127.0.0.1:47862 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
I20260810 22:38:47.732771 10527 catalog_manager.cc:1259] Loaded cluster ID: 4b445ebfd2ca428ebb2131fe655ee0b2
I20260810 22:38:47.732899 10527 catalog_manager.cc:1552] T 00000000000000000000000000000000 P eff5a2103e23408c8d6eac602958b675: loading cluster ID for follower catalog manager: success
I20260810 22:38:47.734603 10527 catalog_manager.cc:1574] T 00000000000000000000000000000000 P eff5a2103e23408c8d6eac602958b675: acquiring CA information for follower catalog manager: success
I20260810 22:38:47.735971 10527 catalog_manager.cc:1602] T 00000000000000000000000000000000 P eff5a2103e23408c8d6eac602958b675: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 22:38:47.762928 10559 catalog_manager.cc:1259] Loaded cluster ID: 4b445ebfd2ca428ebb2131fe655ee0b2
I20260810 22:38:47.763060 10559 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cadb66c9079346e6bf77d976055f22f3: loading cluster ID for follower catalog manager: success
I20260810 22:38:47.766003 10559 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cadb66c9079346e6bf77d976055f22f3: acquiring CA information for follower catalog manager: success
I20260810 22:38:47.768123 10559 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cadb66c9079346e6bf77d976055f22f3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 22:38:48.013928 10652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:48.009010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47874 (local address 127.16.233.125:44069)
0810 22:38:48.009357 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:48.009361 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:48.009456 (+ 95us) server_negotiation.cc:407] Connection header received
0810 22:38:48.009604 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:48.009606 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:48.009742 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:48.009925 (+ 183us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:38:48.010307 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:48.010946 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:48.012010 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:48.012246 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:48.012338 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:48.012733 (+ 395us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:38:48.013336 (+ 603us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:38:48.013439 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:48.013579 (+ 140us) server_negotiation.cc:299] Negotiation successful
0810 22:38:48.013639 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":157,"threads_started":1}
I20260810 22:38:48.022642 10659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:48.017882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58680 (local address 127.16.233.124:40229)
0810 22:38:48.018151 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:48.018155 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:48.018170 (+ 15us) server_negotiation.cc:407] Connection header received
0810 22:38:48.018261 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:48.018264 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:48.018394 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:48.018507 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:38:48.018952 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:48.019609 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:48.020899 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:48.021147 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:48.021264 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:48.021624 (+ 360us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:38:48.022164 (+ 540us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:38:48.022268 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:48.022433 (+ 165us) server_negotiation.cc:299] Negotiation successful
0810 22:38:48.022494 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":111,"threads_started":1}
I20260810 22:38:48.032053 10652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:48.027074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47886 (local address 127.16.233.125:44069)
0810 22:38:48.027219 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:48.027223 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:48.027669 (+ 446us) server_negotiation.cc:407] Connection header received
0810 22:38:48.027899 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:48.027903 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:48.027963 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:48.028056 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:38:48.028530 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:48.029243 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:48.030692 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:48.030998 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:48.031082 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:48.031479 (+ 397us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:38:48.031621 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:38:48.031692 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:48.031821 (+ 129us) server_negotiation.cc:299] Negotiation successful
0810 22:38:48.031884 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 22:38:48.032116 10667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:48.026582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53524 (local address 127.16.233.126:36533)
0810 22:38:48.027163 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:48.027169 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:48.027189 (+ 20us) server_negotiation.cc:407] Connection header received
0810 22:38:48.027329 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:48.027335 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:48.027535 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:48.027689 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:38:48.027976 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:48.028900 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:48.030049 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:48.030367 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:48.030519 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:48.030897 (+ 378us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:38:48.031301 (+ 404us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:38:48.031488 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:48.031791 (+ 303us) server_negotiation.cc:299] Negotiation successful
0810 22:38:48.031912 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":90,"threads_started":1}
I20260810 22:38:48.032392 10659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:38:48.027058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58692 (local address 127.16.233.124:40229)
0810 22:38:48.027172 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0810 22:38:48.027178 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:38:48.028043 (+ 865us) server_negotiation.cc:407] Connection header received
0810 22:38:48.028129 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:38:48.028131 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:38:48.028191 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:38:48.028269 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:38:48.028581 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:48.029272 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:48.031143 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:38:48.031473 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:38:48.031588 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:38:48.031862 (+ 274us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:38:48.031945 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:38:48.031994 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 22:38:48.032181 (+ 187us) server_negotiation.cc:299] Negotiation successful
0810 22:38:48.032262 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 22:38:48.036514 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 10258
I20260810 22:38:48.048084 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 10344
I20260810 22:38:48.059093 17317 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfcURSo/build/debug/bin/kudu with pid 10426
2026-08-10T22:38:48Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3443 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (7116 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (312619 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST